Month: September 2017

# list the contents
tar -tvf archive.tar
tar -tvfj archive.tar.bz2

# extract a directory
tar xjfv a8.droot.20170827.tar.bz root/etc/mail

USING -threads 4 messed me up, don’t do it!

# this works to transcode to a webm vp8 file
ffmpeg -i fireworks.mp4 \
-c:v libvpx -crf 10 -b:v 512K \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-pass 1 -y fireworks.webm; \
ffmpeg -i fireworks.mp4 \
-c:v libvpx -crf 10 -b:v 512K \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-pass 2 -y fireworks.webm

# just stream
ffmpeg -i fireworks.mp4 \
-vcodec copy -an -f rtp rtp://10.11.1.96:5122 \
-vn -f rtp rtp://10.11.1.96:5124

# let’s try to stream and transcode
# worked but choppy
ffmpeg -i people.mp4 \
-c:v libvpx -crf 10 -b:v 256K \
-an -f rtp rtp://10.11.1.96:5124 \
-c:a libopus -b:a 64k \
-vn -f rtp rtp://10.11.1.96:5122

sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update

(2017/12/20 had trouble streaming to janus, rolled my own https://gist.github.com/faleev/3435377)

Supporting Chrome and Firefox
Mozilla compatibility page
Chrome blog about vp9
WebM vp9 is cool. Send the full stream to one guy and he can decide to send a lower bitrate version to the next guy with no transcoding!

ffmpeg converting to webm
mozillas bitrate discussion

We’ll do constrained quality, which should provide 3 bitrates in one file.

Let’s target these two bitrates for our old/new (when we upgrade to HD) videos
640x480p @ 24,25,30 512 (LQ), 750 (MQ) 256 (LQ) 375 (MQ) 742 (LQ) 1088 (MQ)
1920x1080p @ 50,60 3000 1500 4350

only a7 worked, a8 gave “Unrecognized option ’tile-columns'”

converting to webm

fight to get ffmpeg installed
mac
brew install libvpx
brew install ffmpeg –with-libvorbis -with-vorbis –with-theora \
–with-libass \
–with-openssl \
–with-libvpx –with-libvpx –with-opus

ffmpeg -i 34-01-05_She_Wronged_Him_Right_512kb.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx-vp9 -c:a libopus \
-pass 1 -speed 4 34-01-05_She_Wronged_Him_Right_512kb.webm && \
ffmpeg -i 34-01-05_She_Wronged_Him_Right_512kb.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx-vp9 -c:a libopus \
-pass 2 -speed 4 -y 34-01-05_She_Wronged_Him_Right_512kb.webm

try to constrain opus to 128bit audio
ffmpeg -i in.mp4 -c:a libopus -vn -vbr on out.opus
ffmpeg -i input -acodec libopus -b:a bitrate -vbr on -compression_level 10 output

ffmpeg -i Betty_Boop_Baby_Be_Good_1935_512kb.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 -passlogfile xxx \
-quality good -crf 34 -c:v libvpx-vp9 \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-pass 1 -speed 4 Betty_Boop_Baby_Be_Good_1935_512kb.webm && \
ffmpeg -i Betty_Boop_Baby_Be_Good_1935_512kb.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 -passlogfile xxx \
-quality good -crf 34 -c:v libvpx-vp9 \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-pass 2 -speed 4 -y Betty_Boop_Baby_Be_Good_1935_512kb.webm

ffmpeg -i people.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx-vp9 -c:a libopus \
-pass 1 -speed 4 people.webm && \
ffmpeg -i people.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx-vp9 -c:a libopus \
-pass 2 -speed 4 -y people.webm

# WHOPSIE janus doesn’t do vp9 yet
ffmpeg -i fireworks.mp4 \
-c:v libvpx -crf 10 -b:v 512K \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-pass 1 -y fireworks.webm; \
ffmpeg -i fireworks.mp4 \
-c:v libvpx -crf 10 -b:v 512K \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-pass 2 -y fireworks.webm

ffmpeg -i input.mp4 -c:v libvpx -qmin 0 -qmax 50 -crf 5 -b:v 1M -c:a libvorbis output.webm
ffmpeg -i input.mp4 -c:v libvpx-vp9 -minrate 5M -maxrate 5M -b:v 5M output.webm

2017/12/21 this worked!
ffmpeg -i fireworks.mp4 \
-c:v libvpx -crf 10 -b:v 512K \
-c:a libopus -b:a 128k -vbr on -compression_level 10 \
-y fc.vp8.webm

ffmpeg -i people.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx -c:a libopus \
-pass 1 -speed 4 people.vp8.webm && \
ffmpeg -i people.mp4 -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx -c:a libopus \
-pass 2 -speed 4 -y people.vp8.webm

ffmpeg -i /usr5/videos/archive.org/GhostsontheLoose/GhostsontheLoose.ogv -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx-vp9 -c:a libopus \
-pass 1 -speed 4 /usr3/webm/archive.org/GhostsontheLoose//GhostsontheLoose.webm && \
ffmpeg -i /usr5/videos/archive.org/GhostsontheLoose/GhostsontheLoose.ogv -vf scale=640×480 -b:v 512k \
-minrate 256k -maxrate 742k -tile-columns 1 -g 240 -threads 4 \
-quality good -crf 34 -c:v libvpx-vp9 -c:a libopus \
-pass 2 -speed 4 -y /usr3/webm/archive.org/GhostsontheLoose//GhostsontheLoose.webm

debian 8 Jessie, had to roll compile ffmpeg myself
apt-get -y install autoconf automake build-essential libass-dev libfreetype6-dev \
libsdl2-dev libtheora-dev libtool libva-dev libvdpau-dev libvorbis-dev libxcb1-dev libxcb-shm0-dev \
libxcb-xfixes0-dev pkg-config texinfo wget zlib1g-dev \
libvpx4 libopus-dev libx264-dev libx265-dev libfdk-aac1 x265 x264
grab some sox stuff too, we’ll need it later
apt-get install libsox-dev sox
./configure \
–prefix=”/usr/local/bin” \
–pkg-config-flags=”–static” \
–enable-gpl \
–enable-libass \
–enable-libfreetype \
–enable-libmp3lame \
–enable-libopus \
–enable-libtheora \
–enable-libvorbis \
–enable-libvpx \
–enable-libx264 \
–enable-libopus \
–enable-nonfree
and it still don’t work for some conversions:
/tmp/buildd/libvpx-1.3.0/vp9/encoder/vp9_encodeframe.c:1747: rd_pick_partition: Assertion `tp_orig < *tp’ failed.
and then these on some archive.org files:
[aac @ 0x3856d60] SSR is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

and on a7, tried to compile from git

converting to webm with watermark

ffmpeg -codecs
Codecs:
D….. = Decoding supported
.E…. = Encoding supported
..V… = Video codec
..A… = Audio codec
..S… = Subtitle codec
…I.. = Intra frame-only codec
….L. = Lossy compression
…..S = Lossless compression
——-

ffmpeg -formats
File formats:
D. = Demuxing supported
.E = Muxing supported

a8 shows
formats:
D matroska,webm Matroska / WebM
E webm WebM
E webm_chunk WebM Chunk Muxer
DE webm_dash_manifest WebM DASH Manifest
codecs:
D.A.L. opus Opus (Opus Interactive Audio Codec)
D.V.L. vp9 Google VP9

a7 shows
formats:
D matroska,webm Matroska / WebM
E webm WebM
E webm_chunk WebM Chunk Muxer
DE webm_dash_manifest WebM DASH Manifest

codecs:
DEV.L. vp9 Google VP9 (decoders: vp9 libvpx-vp9 vp9_cuvid ) (encoders: libvpx-vp9 )
DEA.L. opus Opus (Opus Interactive Audio Codec) (decoders: opus libopus ) (encoders: opus libopus )

ffmpeg -i input -f rtsp -rtsp_transport tcp rtsp://localhost:8888/live.sdp
-c:v libvpx

ffmpeg -i sample.mp4 -strict -2 -f rtsp rtsp://192.168.0.3:1935/live/myStream

ffmpeg -stream_loop -1 -thread_queue_size 4 -i NTSCTVTestPattern.mp4 -strict -2 -vcodec copy -an -f rtp rtp://10.10.1.97:6005 -acodec copy -vn -sdp_file saved_sdp_file -f rtp rtp://10.11.1.97:7005

ffmpeg -stream_loop -1 -thread_queue_size 4 -i NTSCTVTestPattern.mp4 -strict -2 -f rtsp rtsp://10.10.1.97:6005/live/stream

ffmpeg -re -stream_loop -1 -thread_queue_size 4 -i NTSCTVTestPattern.mp4 -strict -2 \
-sdp_file saved_sdp_file \
-f rtsp rtsp://10.10.1.97:6005 \

ffmpeg -protocol_whitelist “file,rtp,udp” -i saved_sdp_file -strict -2 saved_video_file.mp4

Broken file, keyframe not correctly marked
trying to resolve with latest ffmpeg
didn’t help:
ffmpeg version N-89073-gff8f40a Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.5) 20160609

10 tiers working around the world – US east, US west, Australia south east, Europe north, India west. 10th tier was at 2 second latency, way better than I expected after P2Ping around the world. Have a look:

Now I need Big Data to help me figure out who gets the next stream.

to create a local & remote repo

on a0 in /usr3/phomlish/git create RepoName directory
mkdir RepoName
cd RepoName
git init –bare

phomlish@a0:/usr3/phomlish/git/jukebox$ git init –bare
Initialized empty Git repository in /usr3/phomlish/git/jukebox/

****

in working directory (like on mac)
go where the code exists
w01:jukebox phomlish$ pwd
/Users/phomlish/go/src/jukebox

git init
git add -A
git commit -m init

git remote add origin ssh://a0:2222/usr3/phomlish/git/jukebox
git push –set-upstream origin master
git push

****

in working directory (like on mac)
make sure you have a .gitignore
git clone ssh://a0:2222/usr3/phomlish/git/com.mydetv
git add -A
git commit
git push –set-upstream origin master

git status
git log
# which checkout we’re on
git show

git checkout

Create the branch on your local machine and switch in this branch :
git checkout -b [name_of_your_new_branch]
git checkout -b dev

Change working branch :
git checkout [name_of_your_new_branch]
git checkout master

git push
git push origin [name_of_your_new_branch]

# let’s have 2 remotes!
# on a0
phomlish@a0:/usr3/phomlish/git/mydetv-swarm$ git init –bare
Initialized empty Git repository in /usr3/phomlish/git/mydetv-swarm/
# on local
git remote add origin ssh://a0:2222/usr3/phomlish/git/mydetv-swarm

# merging!!!
# from dev to master locally
git checkout master
git merge dev
git push origin master

Milestones
authenticating to Google, Facebook, and Twitter
Chats are working, emoticons, and chat history
bandwidth tests are working, chaining videos is still a bit flaky.

Next Steps
UI work
*adding channel participants to left panel
*adding user controls. edit nickname
*show username for debugging

Next Up
Add gstreamer so tier 1 streams can come from java and not the browser

Ever since I created a job that was responsible for billing, I realized how important timezones and daylight savings are to application development. I had scheduled the job to run at 1:30 AM. Once a year it didn’t run (when daylight savings skipped 1:30 and jumped to 2:30), and once a year it ran twice (when daylight savings jumped back at 2 AM to 1 AM). Then I had to worry about global deployments and learned even more. Now my rule is, do everything in UTC. Convert things for users to display in their timezone, but store it all UTC.

Some notes for Java 8:

import java.time.ZonedDateTime;
converting:
ZonedDateTime zdt = ZonedDateTime.parse(input, fmt);
Timestamp sqlTs = Timestamp.from(zdt.toInstant());

ZonedDateTime zonedDateTime = ZonedDateTime.now();

ZoneId zoneId = ZoneId.of(“UTC+1”);

ZonedDateTime zonedDateTime2 =
ZonedDateTime.of(2015, 11, 30, 23, 45, 59, 1234, zoneId);
ZonedDateTime newZoneDateTime =
previousDateTime.plus(Period.ofDays(3));
ZoneId zoneId = ZoneId.of(“UTC+1”);
ZoneId zoneId2 = ZoneId.of(“Europe/Copenhagen”);

Some notes for mysql and joda
mysql has a field time datetime which is actually the number of seconds since 1970
it includes milliseconds as a fractional part.
joda has a datetime which is similar
for sprocs, pass datetime as timestamp

to convert from joda datetime to mysql’s timestamp
java.sql.Timestamp d = new java.sql.Timestamp(jodaDateTime.getMillis());

and back again
os.setDt(new DateTime(java.sql.Timestamp.valueOf(rs.getTimestamp(“dt”).toString())));

that worked but I’m hoping for a better way

Who knew there was so much to know about a Favicon. A Cheat Sheet to get things straight and a generator to get the job done.

Let’s define what we want for our websites.

Firefox, Chrome, IE, Edge and Devices IOS and Android.
Animations are just plain irritating so none of that.

Establishing a lowest common denominator:
16×16 and 32×32
IOS: 70×76

facebook authentication screen will use 1024×1024

Shorten is an old format to create lossless compressed audio files. FLAC is the new way. I’ll show some of the joy I’ve had trying to convert the old shn files to flac.

ffmpeg seems to want to do it if you have it compiled with the correct libraries.
ffmpeg -i file.shn file.shn.flac

A java signaling server for connecting up the various clients.

Authentication will be done via major providers, users can pick their favorite- facebook, google, twitter, microsoft, linkedin. This will allow us to use their profile picture, email address, and if they are greater that 18 from the provider.

Websockets is used for the signaling to/from the server.

Webrtc will be used for the audio/video and data streams. Data streams will be used to test the users bandwidth (even though getStats is broken for data streams so we’ll write our own).

From the start we will accommodate just two channels- 24×7 reruns and live. Each channel will serve two streams: standard and low quality (for users with limited bandwidth or data caps imposed by their ISP). Standard quality will be around 1.5 to 3 Mbps and low quality will be 500 Kbps.

Tier zero is the original stream. Tier 1 is hosted at My Delaware TV and will serve several tier 2 users. Tier 2 users will serve tier 3 users, and so on down the pyramid. Testing will provide the practical number of users each tier can serve, all the way down to tier n.

Say tier 1 will serve 4 tier 2 users. Tier 2 will each serve two tier 3 users, allowing us to accommodate 4+4*2= 12 total users. Tier 10 will be able to accommodate 2044 users.

tier users in tier
2 4
3 12
4 28
5 60
6 124
7 252
8 508
9 1020
10 2044

To directly serve that many users a 2 Mbps feed My Delaware TV would need 4088 Mpbs. At Verizon business FIOS pricing using my current bill/speed, I would need 28 150 Mbps lines at a cost of $4,140/month.