How to get audio language from PMT?

[fpgamaster]

Hi All,

When input is MPEG TS over UDP, the language information for audio streams could be available in PMT.
Is there a way to use this information right now?

Regards

[kqyang]

@fpgamaster Are you referring to possible "ISO639 Language Descriptor" in PMT? That is not parsed right now. Do you have a sample content with this descriptor?

Also, we love contributions. If you'd like to contribute, here is the places you'll need to modify:

https://github.com/google/shaka-packager/blob/82e0fd2095632e19ef377a89721bd0d99d90e17e/packager/media/formats/mp2t/ts_section_pmt.cc#L72

https://github.com/google/shaka-packager/blob/82e0fd2095632e19ef377a89721bd0d99d90e17e/packager/media/formats/mp2t/mp2t_media_parser.cc#L274

[fpgamaster]

@kqyang Yes, I am referring to "ISO639 Language Descriptor". A sample content could be obtained from any DVB C/T/S network. I will try to record a sample content with multiple audios and will post a link in few days.
BTW: ffmpeg should generate DVB tables when the output is mpegts and maybe using e.g. '-metadata:s:a:0 language=eng' as a option will do the trick.

[fpgamaster]

@kqyang Feel free to modify the following patch. It is really a quick and dirty solution for the functionality we currently need. Enjoy!

diff --git a/packager/media/formats/mp2t/es_parser_audio.cc b/packager/media/formats/mp2t/es_parser_audio.cc
index 9599563d73..eec0a2dab4 100644
--- a/packager/media/formats/mp2t/es_parser_audio.cc
+++ b/packager/media/formats/mp2t/es_parser_audio.cc
@@ -80,11 +80,13 @@ static bool LookForSyncWord(const uint8_t* raw_es,
 
 EsParserAudio::EsParserAudio(uint32_t pid,
                              TsStreamType stream_type,
+                             std::string lang,
                              const NewStreamInfoCB& new_stream_info_cb,
                              const EmitSampleCB& emit_sample_cb,
                              bool sbr_in_mimetype)
     : EsParser(pid),
       stream_type_(stream_type),
+      lang_(lang),
       new_stream_info_cb_(new_stream_info_cb),
       emit_sample_cb_(emit_sample_cb),
       sbr_in_mimetype_(sbr_in_mimetype) {
@@ -213,7 +215,7 @@ bool EsParserAudio::UpdateAudioConfiguration(const AudioHeader& audio_header) {
       audio_specific_config.data(), audio_specific_config.size(),
       kAacSampleSizeBits, audio_header.GetNumChannels(),
       extended_samples_per_second, 0 /* seek preroll */, 0 /* codec delay */,
-      0 /* max bitrate */, 0 /* avg bitrate */, std::string(), false);
+      0 /* max bitrate */, 0 /* avg bitrate */, lang_, false);
 
   DVLOG(1) << "Sampling frequency: " << samples_per_second;
   DVLOG(1) << "Extended sampling frequency: " << extended_samples_per_second;
diff --git a/packager/media/formats/mp2t/es_parser_audio.h b/packager/media/formats/mp2t/es_parser_audio.h
index e92a9fefb0..69b1ae2375 100644
--- a/packager/media/formats/mp2t/es_parser_audio.h
+++ b/packager/media/formats/mp2t/es_parser_audio.h
@@ -29,6 +29,7 @@ class EsParserAudio : public EsParser {
  public:
   EsParserAudio(uint32_t pid,
                 TsStreamType stream_type,
+                std::string lang,
                 const NewStreamInfoCB& new_stream_info_cb,
                 const EmitSampleCB& emit_sample_cb,
                 bool sbr_in_mimetype);
@@ -57,6 +58,8 @@ class EsParserAudio : public EsParser {
   const TsStreamType stream_type_;
   std::unique_ptr<AudioHeader> audio_header_;
 
+  std::string lang_;
+
   // Callbacks:
   // - to signal a new audio configuration,
   // - to send ES buffers.
diff --git a/packager/media/formats/mp2t/mp2t_media_parser.cc b/packager/media/formats/mp2t/mp2t_media_parser.cc
index 4ecb9dd76a..acd95e89f2 100644
--- a/packager/media/formats/mp2t/mp2t_media_parser.cc
+++ b/packager/media/formats/mp2t/mp2t_media_parser.cc
@@ -273,10 +273,12 @@ void Mp2tMediaParser::RegisterPmt(int program_number, int pmt_pid) {
 
 void Mp2tMediaParser::RegisterPes(int pmt_pid,
                                   int pes_pid,
-                                  int stream_type) {
+                                  int stream_type,
+                                  std::string lang) {
   DVLOG(1) << "RegisterPes:"
            << " pes_pid=" << pes_pid
-           << " stream_type=" << std::hex << stream_type << std::dec;
+           << " stream_type=" << std::hex << stream_type << std::dec
+           << " lang=" << lang;
   std::map<int, std::unique_ptr<PidState>>::iterator it = pids_.find(pes_pid);
   if (it != pids_.end())
     return;
@@ -300,7 +302,7 @@ void Mp2tMediaParser::RegisterPes(int pmt_pid,
     case TsStreamType::kAdtsAac:
     case TsStreamType::kAc3:
       es_parser.reset(new EsParserAudio(
-          pes_pid, static_cast<TsStreamType>(stream_type),
+          pes_pid, static_cast<TsStreamType>(stream_type), lang,
           base::Bind(&Mp2tMediaParser::OnNewStreamInfo, base::Unretained(this)),
           base::Bind(&Mp2tMediaParser::OnEmitSample, base::Unretained(this)),
           sbr_in_mimetype_));
diff --git a/packager/media/formats/mp2t/mp2t_media_parser.h b/packager/media/formats/mp2t/mp2t_media_parser.h
index 2ad7fbd733..ed6605fff7 100644
--- a/packager/media/formats/mp2t/mp2t_media_parser.h
+++ b/packager/media/formats/mp2t/mp2t_media_parser.h
@@ -51,7 +51,7 @@ class Mp2tMediaParser : public MediaParser {
   // Possible values for |media_type| are defined in:
   // ISO-13818.1 / ITU H.222 Table 2.34 "Media type assignments".
   // |pes_pid| is part of the Program Map Table refered by |pmt_pid|.
-  void RegisterPes(int pmt_pid, int pes_pid, int media_type);
+  void RegisterPes(int pmt_pid, int pes_pid, int media_type, std::string lang);
 
   // Callback invoked each time the audio/video decoder configuration is
   // changed.
diff --git a/packager/media/formats/mp2t/ts_section_pmt.cc b/packager/media/formats/mp2t/ts_section_pmt.cc
index 8ba854cb2e..559d7ade75 100644
--- a/packager/media/formats/mp2t/ts_section_pmt.cc
+++ b/packager/media/formats/mp2t/ts_section_pmt.cc
@@ -76,6 +76,7 @@ bool TsSectionPmt::ParsePsiSection(BitReader* bit_reader) {
   // (4 bytes = size of the CRC).
   int pid_map_end_marker = section_start_marker - section_length + 4;
   std::map<int, int> pid_map;
+  std::map<int, std::string> pid_lang;
   while (static_cast<int>(bit_reader->bits_available()) >
          8 * pid_map_end_marker) {
     int stream_type;
@@ -95,7 +96,29 @@ bool TsSectionPmt::ParsePsiSection(BitReader* bit_reader) {
 
     // Read the ES info descriptors.
     // Defined in section 2.6 of ISO-13818.
-    RCHECK(bit_reader->SkipBits(8 * es_info_length));
+    int descriptor_tag;
+    int descriptor_length;
+    char lang[] = { 'u', 'n', 'd', '\0' };
+
+    while (es_info_length) {
+	RCHECK(bit_reader->ReadBits(8, &descriptor_tag));
+	RCHECK(bit_reader->ReadBits(8, &descriptor_length));
+
+	if (descriptor_tag == 10) {
+		RCHECK(bit_reader->ReadBits(8, &lang[0]));	// ISO_639_language_code
+		RCHECK(bit_reader->ReadBits(8, &lang[1]));
+		RCHECK(bit_reader->ReadBits(8, &lang[2]));
+		RCHECK(bit_reader->SkipBits(8));		// audio_type
+
+	}
+	else {
+		RCHECK(bit_reader->SkipBits(8 * descriptor_length));
+	}
+
+	es_info_length -= 2 + descriptor_length;
+    }
+    pid_lang.insert(std::pair<int, std::string>(pid_es, std::string(lang)));
+
   }
 
   // Read the CRC.
@@ -105,7 +128,7 @@ bool TsSectionPmt::ParsePsiSection(BitReader* bit_reader) {
   // Once the PMT has been proved to be correct, register the PIDs.
   for (std::map<int, int>::iterator it = pid_map.begin();
        it != pid_map.end(); ++it)
-    register_pes_cb_.Run(it->first, it->second);
+    register_pes_cb_.Run(it->first, it->second, pid_lang[it->first]);
 
   return true;
 }
diff --git a/packager/media/formats/mp2t/ts_section_pmt.h b/packager/media/formats/mp2t/ts_section_pmt.h
index 7b14d82b70..8e1a5ade6d 100644
--- a/packager/media/formats/mp2t/ts_section_pmt.h
+++ b/packager/media/formats/mp2t/ts_section_pmt.h
@@ -18,7 +18,7 @@ class TsSectionPmt : public TsSectionPsi {
   // RegisterPesCb::Run(int pes_pid, int stream_type);
   // Stream type is defined in
   // "Table 2-34 – Stream type assignments" in H.222
-  typedef base::Callback<void(int, int)> RegisterPesCb;
+  typedef base::Callback<void(int, int, std::string)> RegisterPesCb;
 
   explicit TsSectionPmt(const RegisterPesCb& register_pes_cb);
   ~TsSectionPmt() override;

[kqyang]

@fpgamaster Thanks for the patch. It is quite good I think, with only a few minor nits. Do you want to submit a pull request for this?

[fpgamaster]

@kqyang Yes, it will be nice!

[kqyang]

@fpgamaster Do you still plan to submit a pull request for this feature?

Thanks.

[kqyang]

@sammirata volunteers to pick up this issue, see #518. Thank you @sammirata. Appreciated!

[modernletter]

I would like to pick and extend this issue. PMT can be source not only for audio language, but also other metadata:

• language for other types of stream (e.g. subtitles, teletext) (
• audio type (a.k.a. audio description, but also clean audio)
• maximum bitrate - which is not very interesting, but better than default 0 provided by EsParserAudio

For example this is part of tstable output with stream containing these descriptors (ttable is part of tsduck suite, very useful tool to analyse/modify transport streams :

    Elementary stream: type 0x0F (MPEG-2 AAC Audio), PID: 484 (0x01E4)
    - Descriptor 0: ISO-639 Language (0x0A, 10), 4 bytes
      Language: pol, Type: 0x00 (undefined)
    - Descriptor 1: Maximum Bitrate (0x0E, 14), 3 bytes
      Maximum bitrate: 0x000001BB (443), 177,200 bits/second
    Elementary stream: type 0x0F (MPEG-2 AAC Audio), PID: 485 (0x01E5)
    - Descriptor 0: ISO-639 Language (0x0A, 10), 4 bytes
      Language: mis, Type: 0x03 (visual impaired commentary)
    - Descriptor 1: Maximum Bitrate (0x0E, 14), 3 bytes
      Maximum bitrate: 0x000001BB (443), 177,200 bits/second
    Elementary stream: type 0x86 (SCTE 35 Splice Info), PID: 500 (0x01F4)
    Elementary stream: type 0x06 (MPEG-2 PES private data), PID: 32 (0x0020)
    - Descriptor 0: Teletext (0x56, 86), 10 bytes
      Language: pol, Type: 1 (0x01)
      Type: Initial Teletext page
      Magazine: 1, page: 0, full page: 100
      Language: pol, Type: 2 (0x02)
      Type: Teletext subtitles
      Magazine: 7, page: 119, full page: 777
    - Descriptor 1: Maximum Bitrate (0x0E, 14), 3 bytes
      Maximum bitrate: 0x00000235 (565), 226,000 bits/second

I would like to propose a PR to parse these descriptors, with test vector.

Currently there is no field for audio type in AudioStreamInfo. If added, it could be used for e.g. auto-set MPEG-DASH role/accessibility elements or HLS media characteristics attribute. Such field should have some abstract values, to not be bounded with MPEG-TS audio type. If that can be agreed, I would also do that.