Sam Lee
1
COW Coin
Forum Replies Created
-
For data management, the good old 2010-2012 tower Mac Pros served me quite well.
Long-term offline archival, I use LTO-6. Probably will skip LTO-7 and upgrade to LTO-6 several years later. In the long run, LTO-6s are much cheaper than hdds ( high initial tape drive cost but very low tape media costs, power consumption, maintenence-free,..). The problem with hdds is that it will not spin after ## years. HDDs are more for short-term, near and online use. I still have footage back in 1997. All recently archived to LTO-6. Hopefully it can last another 20-30 years.
-
The pristine P2 media is always archived. I never delete them since 2006. It’s filled with full P2 metadata and even GPS coordinates from the 2/3″ cams. Far too valuable to be deleted. FCP X only takes 4 fields out of 50 possible metadata fields. If there is ever a need for P2 to be on PC, your best shot is to give them the raw P2 media vs the rewrapped .MOV. I can always use Panasonic P2 CMS to fully search the raw asset. That can only be when the pristine P2 media is online and not the rewrapped.
Replacing native P2 with new clip is a challenge. This is another limitation with FCP X. I also work on consumer AVCHD cams once a while and the biggest problem with FCP X is that if you don’t copy the AVCHD media to FCPX’s library (checking the Leave files in place during import), relinking it is virtually impossible when you migrate the media to another hdd or retrieve the media (from LTO-6 archive) at a later date for revision. It’s so hard to do that what I did was to transcode the AVCHD in FCP 7 and it worked! Luckily FCP 7 also supports AVCHD and the exact clip name. I don’t think is in native format but a transcoded Pro Res 422. AVCHD has an annoying generic Clip #01, Clip #02…. I don’t know if you can change it in the camera to something more unique. This is a huge problem when you have dozen of similar media and you may get confused with Clip #01 on Day 2 for Clip #01 on Day 5.
I don’t recall exactly what I did in the past in regard to replacing FCP X’s native P2 import to FCP 7 rewrapped AVC-I 100 .mov. It was something like taking the auto backup of the current library. I had to completely remove all raw P2 assets by disconnecting the drive or shutting it off. Then when that auto backup library is opened, it then asked for relink. This is where FCP X will not relink your original P2 media (unless when if you check “Copy to Library” during import)! I was livid. You should try this at your leisure. What I had to do was go to FCP 7 and rewrap all P2 media. Then I’m able to relink to the newly created rewrapped .mov. At this point, the library is new and it’s much smaller.
At this point, it seems like you have to consolidate into FCP X’s library for a self-contained final archive. This gets expensive fast (storage wise) when there are about 15 Tb of raw P2 data. By rewrapping P2 media in FCP 7, I can get the best of both worlds. This of course will not be applicable to every scenario. Nevertheless, for large project, the rewrapping method offers much more flexibility for relinking and media sharing with After Effects.
In FCP 7’s Log & Transfer, the P2 folder name will be clearly indicated on the top. If you have a long folder, simply drag the window a bit more to see the full names vs being truncated by other fields. It will tell you if you spanning is complete or incomplete. Very nice but unfortunately none of these features get ported to FCP X.
-
Set your FCP 7’s scratch disk to the drive you want the media to be rewrapped. Go to Log &Transfer. Import all P2 folders by single or compound folder slection. Look for Complete span clip status (if P2 media are spanned). Click “Add Selection to Queue” to begin the rewrapping. When all done rename the re-wrapped .MOV. Move & rename to correct folder for FCP X. In FCP X, simply import all of it. It’s quite straightforward.
This intermediate step will save gigs to tbs of space when the project grows. FCP X is just not efficient when it comes to handling native P2 codecs in general.
-
Panasonic’s P2 rich metadata gets cut out in FCP 7 & X. Only camera model, serial #, clip name, & shot marker gets imported in FCP X. With that in mind, the clip name is only useful. I do lots of unscripted docs and have been using generic names. The most useful for me is separating each camera. It’s virtually impossible to sort of six cameras using P2’s default clip name. You can update the metadata via the SDHC on the HPX-250 or go in and enter it manually. It’s a bit cumbersome in the camera but it works just fine. I typically would increment the shoot day clip name in the camera. The only downside to this is that you have to remember to do it everyday. If you forget, that clip name metadata data will forever be stuck. To avoid mistakes, I keep it generic as much as possible these days.
Regarding rewrapping and transcoding in FCP 7: I’d strongly recommend to rewrap the file. That’s the default codec handling setting when installing FCP 7. Transcoding to Pro Res 422 from AVC-I takes a lot of CPU time and a bit more hdd space per file. When you have a large raw footage asset, the space adds up to Tb with the transcoded Pro Res 422 or HQ. There’s no setting in FCP X to treat the P2 AVC-I 100 media as native or transcode to Pro Res 422 or HQ. Plus the file size for AVC-I 100 & Pro Res 422 .MOV: Pro Res 422 is a bit larger. This gets expensive when you have a big library. You cannot transcode directly from Compressor. Earlier version of Compressor only takes regular clips and not camera raw. Quality wise, I feel more comfortable working in native AVC-I 100 .MOV codec than Pro Res 422 .MOV. It will preserve all of the gamma and other parameters as closest as possible. The most important thing here is FCP 7 & X take in the rewrapped AVC-I 100 .MOV files just like Pro Res 422 .MOV. You can verify that the P2 AVC-I 100 clips are indeed a rewrapped type by simply go to Quicktime and look at its codec setting. It’ll state AVC-I 100 vs Pro Res 422.
Come to think of it, I have yet to investigate whether FCP X is forcing all P2 formats like a transcoded media Pro Res 422 HQ. I always wondered why it’s so much larger when I import P2 natively in FCP X than when I rewrapped in FCP 7 as a native AVC-I 100 codec. I typically save about 60% of disk space when using FCP 7 rewrapped .mov. That’s a huge space saving for large projects. And quality wise, I can’t tell the difference between the two: AVC-I 100 rewrapped or Pro Res 422 HQ transcoded .movs. Most of these were shot with the HPX-3100s, PX5000s, and few HPX-250s cams. With that in mind, I don’t see a need for Pro Res 422 at the expense of ~60% more disk space needed.
-
If you have been shooting all short clips on the first card, nothing will be spanned of course. But if you happen to be using the last 1 Gb on a 64Gb P2 card #1 and record 20 min non-stop, that last clip will get span over to 64 Gb P2 card #2. There’s no logical reason for the clip to span over when the first card is not filled to the maximum capacity it can handle. For scripted type of shoot with short takes, I typically do not span the card. But unscripted docs, I will span it since it’s unpredictable and don’t want to be bothered with changing media during an important scene that can’t be reshot or recreated.
And if course you can continuously record as much as you like. The longest that I record continuously was 4.5 hours at full AVC-I 100. Every other hour I swap card and it keeps on recording. You really have to be comfortable with the blinking green on the P2 slot to remove it. It can get a bit nervous. But the blinking green and blinking orange works for me every time. Looking back this is a pretty risky thing to do and I would pause the camera as much as I can. What if you accidentally remove the wrong slot with blinking orange (indicating that it’s recording)? I haven’t experience this but I think only the last 5 min will be bad and not the entire duration of the spanned clip.
Those P2 icons can only be generated from the P2 readers (PCD-35, PCD-30) and the AG-MSU10 mobile storage unit. It cannot be created from an internal or external hdd. Those icons are from Panasonic’s genuine gear and they have their own custom volume P2 artwork. But you must install Panasonic’s own free P2 drivers software. It helps a lot to tell where the media originated. Shot Put Pro doesn’t generate icon when the folders are copied. It’s just a simple media copying utility with full verification.
-
FCP X should not break up the spanned clips unless when you import it, a folder is missing. BTW, FCP 7’s Log & Transfer function is a great tool to determine if all of your P2 folders are present for uninterrupted spanning. Sadly, there is no such tool in FCP X. It’s so necessary but just not present in FCP X. Perhaps you may want run it as a spanning evaluator tool but doesn’t have to perform the Log & Transfer function. I actually use FCP 7 as a tool to determine if all the spanned clips from all the cams are present before reformatting all of the P2 cards. It will say immediately give you either spanning is complete or incomplete by adding or removing P2 folders. Be aware that if you have P2 cards mounted, it will go to that first. And if you know for cetain that only 2 P2 cards are shot, then you have to mount 2 of it. If not, eject the P2 drives and custom point it to the two P2 folders closest to \CONTENTS on your hdd instead.
If I’m not mistaken, FCP 7 & X prefer the one closest to the P2 folder. If you put a two folders before it, it may not see the clips. For certain, FCP 7 will give you an invalid P2 dir error right away if there are 2 folders. It only wants a single folder before the P2’s \CONTENTS media). I typically will compound select all the folders right before \CONTENTS and that seems to work everytime.
-
FCP X automates the file transcoding process internally and user can’t change any setting. The three popular 1080pformats: AVCHD, AVC-I (P2), Sony IMX (SxS) will automatically recognized only clip name. It will ignore all other directory attributes that are irrelevant to most editors.
The only way to manually preserve P2’s file structure is by creating a camera archive for each card. Thanks but no thanks when I have 50 cards per day to deal with. Your probably are asking why this is important. It’s because if you ever need to work in PC Windows, the raw P2 media will be Mac/PC compatible otherwise you’ll be stuck to Mac only once it gets rewrapped to .MOV and you delete the native P2 raw media. FCP does not perform a bit for bit data transfer to another hdd for you. That has to be done with OSX Finder or Shot Put Pro.
Manually creating a folder name in OSX Finder is one of the simplest ways to transfer the raw data from P2 and other raw cam formats. A more elegant method is by using 3rd party app called Shot Put Pro. For $100, it’ll automate the process and even verify that the data is 100% error-free. Eventually if you shoot a lot of footage, you will need this util to expedite the offloading process.
At this point in time, I have not been able to find an efficient file rewrapping workflow in FCP X that will result in smaller library size. I contacted Apple and others but no luck. The only viable solution now is to rewrap in FCP 7 and import as a native AVC-I 100 .mov clips. That is the most efficient for FCP X now. Perhaps next year if enough users complain, Apple will revamp their P2 AVC-I support.
-
The top icon with P2 series F are from the PCD-30, PCD-35 P2 drives. It has the slot #s also. Nice little touch. The orange icons are all from the AG-MSU10 drive. Up to 23 P2 drives per AG-MSU10 can be mounted. I’ll only mount to up to 55-60 P2 drives per session. Anything beyond it will crash or freeze up the system.
I’m rewrapping the P2 AVC-I 100 via FCP7. You can say it’s transcoding but not exactly because it’s set to the native AVC-I 100 .MOV. It’ll be transcoding if I set the Log & Transfer to Pro Res 422. You’ll notice a huge space saving when your project is in the Tb range. Recent internal tests clearly shown over for a typical 1 Tb of raw AVC-100 P2 footage direct FCP 10.2.1 native file import takes up about 390 Gb in the library. Optimized media selection was used. Proxy was not used. On the other hand, if I spend a bit more time rewrapping the AVC-I 100 footage via FCP 7 and import it to FCP X for editing, the library is only about 300-400 Mb (with the Leave Files in place checked). With proxy option, the rewrapped .MOV import is only about 120 Gb for the FCP X library. That’s a huge saving in file size in both cases. It adds up when the edit get larger. Whey you happen to be editing large media library, keep this in mind. You do not want a big 12 Tb library. I had one that grown to 8 Tb for native P2 import. I have no idea what FCP X is doing but importing native P2 media is not at all efficient in 10.2.1.
On a side note, I prefer rewrapping to .MOV because other apps – such as After Effects can take that footage with ease. Trying to get AE to read footage from the raw P2 is too time consuming. It’s also cost prohibitive having to duplicate the raw footage (one for FCP X’s library) and the other for the 3rd party apps to use. FCP X’s library for all media is hidden in a .fcpbundle and 3rd party app like AE can’t really use it at all.
-
Here’s a typical weekly workflow for me. I’m ingesting about 40-50 64 Gb P2 cards per day. this is for a 6-cam, unscripted reality show shot on 1/3″ and 2/3″ P2 cams. The key here is that the volume name has to be structured. No mistaken can be made at anytime. That is, I must be 100% certain that all P2 volumes are copied and verified (XXHash64). P2 has an annoying 11 char volume limit. So one must find a way to name it based on the workflow and situation of each prod. The 11-char P2 volume name is foundation name for all other names. It looks somewhat unorganized now, but eventually will be uniformly named.
My folder naming is actually quite generic and general. I simply don’t have the extra time to custom name 40-50 folders per day. Typically it’s categorized by day and camera #. Nothing else otherwise the folder gets too long.
Shot Put Pro 5.3.3 is a must have app. It will take all of the card volumes and transfer it automatically. It’s impractical to manually do each one when you have about 40 P2 cards per day to deal with on a regular basis. You can also specify pre and suffixes volume name. Due to system resource limit, I can only offload about 10-15 cards at a single time. After that, continue on with the 2nd batch of 10 P2 cards. The P2 sources are from PCD-35s, PCD-30, AG-MSU10s, PCD-2. Practically every possible reader I can get my hand is utilized on the legacy Mac Pro 2010 model. I do not like Micro P2s. They are very easy to get lost in the field. Used market cost for Micro P2 and P2 is only ~$50 difference. Sadly, the PX270 and PX380 will force users to buy Micro P2s if they want spanning. Otherwise only a single slot for P2 on those cams.
True metadata of camera name, clip names, etc are done at the camera level and not in FCPX. It’s too cumbersome to manually rename it FCP X, especially when there are about 6-8 cameras to deal with. Time code sync must be correctly done during the shoot via Zaxcom or Time Code buddy products. If there’re drift, it’s very difficult to manually offset TC with the native P2 media. Small thing such as correct date and time setting on each camera is also important as it’ll affect the initial sorting. Overall every detail has to be taken into account in order to effectively manage the massive P2 media on a daily basis.
After extensive real world testings, it’s not efficient to edit P2 natively in FCP X. Library size is about 8-10x larger than the rewrapped P2 files (via FCP 7.0.3 as a native AVC-I MOV). For smaller edit with only a few P2 cards, then it’s OK. But large scale edit, not practical or recommended.
-
I just tried on FCP 10.2.1 and all spanned clips worked fine. Here are my scenarios:
I have a PCD-35 which can read all 5 P2 cards. I happen to have a five 64 Gb P2 cards shoot and it imports spanned clip just fine. The catch here is that you must have all 5 cards online during the import. If there is the 6th card needed it will not span properly. Not to worry. There are shoots where I average about 30-35 cards per day for 6 cameras. This is where you must copy all native P2 to a hdd and then import all at once. Metadata here on the clip name is a must have here, especially with 6 or more cameras. I can’t imagine anybody can sort through the default P2 naming convention.
The true metadata is really the clip name of the footage shot. This has to be done when you shoot on your P2 camcorder. If you don’t set it to record clip name and type 2 meta data, FCP will always use the default $$$$##.MXF instead of the custom “Acam-main.MXF” that you specified in the camera. Otherwise 10.2.1 will make its own keyword based on the folder you import. It’s good to name the P2 folder name as a meaningful name vs something generic.
The 2nd method that I always do is to copy all of my pristine P2 file to the hdd for archive. I’m using Shot Put Pro 5.3.3 and it has the directory as follow: “Aug 2015 Work For Hire Shoots / Aug 28 Reality A cam.. B cam, C cam, D cam. What I did was select the closest one to the P2 structure (/CONTENTS). Shift select all 5 folders and it imports in with fully spanning. The key here is that you must shift select all 5 folders and FCP 10.2.1 will auto span it if it sees it on the original P2 folder. FCP 10.2.1 import window elegantly filter out the P2 structure to just the clip name. That is you will not see other directory name in the P2 (audio, clip, icon, proxy..)There’s absolutely no 5 min orphaned spanning clip. If not all 5 folders are selected it will result in missing spanned clip.
Whatever the case, spanning for FCP 10.2.1 works pretty well.
