Couple notes about the IDs on Alfred, we may want to start a Post here...
Couple notes about the IDs on Alfred, we may want to start a Post here with all this information.
pub-item -> ID (00-dash its owned, none 00- other distr) "internal sku" it splited in Pub# and Item columns too. Company field also used
-zz-Distributed are external Publishers (Ignore zz- folders for now)
-Numeric Items goes to numeric folders 000-xxx (so 24 "0024" will be at 00000-09999/000-0999/24, 4229 is 00000-09999/4000-4999/4229)
-Alpha numeric Items goes to Warner-X (so H1 will be at Warner-H/H1)
Couple notes about the IDs on Alfred, we may want to start a Post here with all this information.
pub-item -> ID (00-dash its owned, none 00- other distr) "internal sku" it splited in Pub# and Item columns too. Company field also used
-zz-Distributed are external Publishers (Ignore zz- folders for now)
-Numeric Items goes to numeric folders 000-xxx (so 24 "0024" will be at 00000-09999/000-0999/24, 4229 is 00000-09999/4000-4999/4229)
-Alpha numeric Items goes to Warner-X (so H1 will be at Warner-H/H1)
Complete server migration and data sync by next week.
Update desktop tool version and coordinate rollout with Travis.
Resolve IT-related delays and finalize data recovery.
Add a column to the spreadsheet to facilitate file mapping.
Improve project documentation and task tracking for ongoing initiatives.
Meeting Summary - Alfred / eMedia Library - October 31
1. Server Migration and Data Consolidation
Update: Christopher shared progress on migrating all files and data to a single "Alfred DB" server, syncing from the old server, Dropbox, and spreadsheet data. Target completion is next week.
2. Desktop Tool Updates and Rollout
Plan: Desktop tool enhancements have been made, and Travis is set to update to the latest version on the new server next week, with close team support for a smooth transition.
3. IT Challenges and Data Recovery
Issue: Transition to a contractor IT team after the head of IT's departure revealed previous issues with server crash handling and data recovery. Efforts are underway to resolve delays.
4. Spreadsheet Data and File Mapping
Challenge: Mapping spreadsheet data to correct files and folders remains an issue. Team considering adding a column in the spreadsheet to streamline file paths.
5. Relationship Management and Versioning
Feature: Smart organizer tool now manages file relationships and versioning, which could improve Alfred’s file organization and workflows.
6. Next Steps
Actions: Continue data consolidation, update desktop tools, support transition for Travis, and implement better task documentation through structured project management.
Action Points
Complete server migration and data sync by next week.
Update desktop tool version and coordinate rollout with Travis.
Resolve IT-related delays and finalize data recovery.
Add a column to the spreadsheet to facilitate file mapping.
Improve project documentation and task tracking for ongoing initiatives.
Release new version if Desktop App into the Apple Play Store
Alfred check invalid Folder names for : skip chars
Alfred Import UI needs indicator panel/div that says what its downloading
TODO: Finish Prior media scan on new Alfred instance and use it for searching
Hey Dieneke,
I don't have that information yet. The reason I haven't addressed it yet is because we're still dealing with some server issues on our end. Our IT Department recently migrated our Mac server (the one that holds these assets) to a PC server. However, we've found a lot of corrupt or missing directories that they are having to fix. My direct supervisor asked me to not spend any time on our DAM until this is done, and unfortunately I don't have an ETA for when our IT Department will finalize the recovery. To put it into perspective, we've been waiting 3-4 months already.
At this point, we're using EnterMedia as a search engine mostly. We search for the files, and then revert to our server to access them. This works well for us right now. However, once our new server is up and running, that's when we'll want to hit the ground running re-organizing our assets. That is when these advancements will be useful.
It might be better to hold off meeting until our IT department has finished their recovery? I want to avoid us doing any development work until our server is back up, reliable, and ready to be sorted through.
Hey Dieneke,
I haven't been able to put this together yet, and in fact, I need to postpone our meetings until the beginning of August. We've had to pivot our priorities and my attention needs to be elsewhere until August.
I apologize for this, but let's resume the meeting in August.
Please let me know if you have any questions/comments/concerns.
Next meeting thursday the 30th
Tuesday/wednesday , Monday is memorial day.
ohh
Travis will get the document for the path early next week.
Meeting today cancelled.
Meeting today cancelled.
desktop client: Add Pull/Push sync
desktop client: Make sure open workspace shows window
desktop client: Try and upgrade all js from our build server? Or from the workspace
desktop client: Rename eMediaWorkspace eMediaLibrary
Import Search: Show what we actually searched on in results
Import Search: Use REST API from old server
Import Search: Show wait icon when searching
App: Disable registration on login
Data: Connect Thumbails from other ftp jpg imports
Data: Consider making other areas such as External Distributions More data is available from filemaker. Is this part of a group
Data: Travis will add a column for Source Path
Data: Travis will ask about starting a download from Dropbox
Hey Travis, we talk yesterday about adding a sourcepath field in the Database on the Ftp, where you can point us the path where the files are stored, this is the file we are reading, please let us know if you can add it: ftp1.alfred.com/CatData/ALF_ALL_ALL_PipeDelimited.txt
alfred 24/4
Look asset if people are working on the same project
Once Alfred is up and running:
How team can suss out the assets and assign them to publications
requires manual workflow.
New version gives that workflow and makes is more easy.
Wants to work with eMedia mainly.
Servers might be out of date, PC server, wants to talk about it
Related products flavors... Digital PDFs for sale. New item numbers for sales reporting. Metadata. The artwork on a disk or mp3 are part of parent but tracked with other items. Internal and External Sellable vs Components
Workflow Need: Update all related items when one component has been updated
Import via REST API and Add pagination Add more search options? Maybe by file types
Do a New rsync and index fresh to Fix missing files issue on old server. Dont full text index
700G out of 18TB
Remove full text
Start ingesting
Hi Travis, we have made some good progress. The dropbox is downloading and it has about 500k files. It is very slow to download but is moving along. We also are testing the smart download and it is going well. We will want to get some feedback from you on that next time we meet.
Alfred is cancelled for this friday l.
Check on servers more.
Get drop box imported
Get Desktop Downloading working
hey! whats the status on the servers of Alfred? still running slow?
Alfred
1. Desktop tool for opening files
2. "Products" -> One to many "Production Assets" (individial files or folders)
2a. Add field to Product Assets table called Production Asset Type (old, primary etc)
3. Add Collaboration Folders
4. Script to append "Production Assets" from Dropbox by Item number in Product DB
5. Add a "Series" field to Products table (to filter by)
Print Prodution Archive Files
By product range
Assets/Product number
22GB
Search old volumes
Copy to old sub directory
Edit them and copy to new location
New SQL DB in the works on premise
Add filters such as Series
Publication Date ( to filter by years)
Harry potter
See web site
Add Collaboration
Add queue for downloads to show folders?
Download only and pick a folder
Script to append Production Folders from Dropbox by Item number
Production Folder maybe a file only... So rename to Production Assets
Add Asset Type such as "Old" or "Primary Folder"
Cristobal, could you check this?
Hey Dieneke,
Has tech@entermediadb.org accepted the invite from our dropbox archive? We aren't seeing it as accepted yet. Can you please let us know once you've accepted the invite so that we can adjust your access privileges? Thanks so much!
Travis C. Haase
Monday:
Exported: 3,942,295
Pending: 2,887,724
Complete: 1,052,958
So we did around 500K metadata-read & conversions during the weekend
Numbers before the weekend:
Exported: 3,942,295
Pending: 3,359,924
Complete: 581,321
If you want to do a debug sesh, I'm free for 3 hours.
Oh good to know, glad I got held off.
we just met Alfted this morning, we may hold on on the filters since we will get new fields from them
ok
credentials are admin/admin
Hi, I'd love to work on those filters for a bit if you'd like, I didn't have the log in and for some reason wasn't able to get ahold of you guys.
Related products
Book only
Book and CD
Same product but has a variant Component Look up table
Kit merging with CD
Add Drop box. Waiting
Filemaker DB has more DB
Called Products
Clean up dashes when searching 00-29 = 0029
Show only parents
Show lose asset
Add rights and royalties tab? With terriory? With contract? (custom app)
YES Add Components/Kits tab by Book# and CD Item Number, Audio. Player
Internal status field 3 char code
Share missing link
Add Primary PDF, Final Audio
Are realated Products actually products?
Numbers today morning:
Imported: 3,942,295
137,000 Completed
3,800,000 Pending metadata
432,339 Conversions tasks, not sure if this is a good parameter to track
Report at the end of the day:
First Hot Folder (Production) imported 1,218,217
28,000 reported as completed
1,198,000 pending metadata
Conversion table is 75,000 records, It is doing that many conversions because looks like we do not have a lot of generated Media, I took a random Asset, listed the generated Media (untouched yet) and the Original folder, Original folder has 29 files, in 2018 we only generated files for 1 of them. (see attachment)
Add download Order queue to the UI. Have it download to the computer once it's ready. Need this for TAE as well.
Rebrand Media Boat -> eMediaLibrary App
Alfed 10/11
- How are you are looking up products now -> make that easier for clean up team
- Going through database they have now -> dropbox database, we want to connect to that. -> share directory with emedia. Read access is all we need.
- we will send an email adress to to connect tech@entermediadb.org, to set up dropbox
- a sample data instead of the dropbox folder could work too, screen full of results, a volume, 20 products -> travis send us
- We will do that in the nxt weeks and update the b-site and next time we meet we can show that to travis. Need to do a new reindex.
- Filter search is critical, add to collection is not used. Emedia is just primarly used as a search tool.
Goal = search files on dropbox database.
Desktop tool -> show in new future
Clean up -> after local server comes mid january
Problem now! Missing files on entermedia -> Files doesnt excist, we need to erase. Fresh scan of the files that we have.
https://zoom.us/j/94713777672?pwd=Q1dhYmNCUld3Rkp2UkEwemY2WEJHdz09
Were in there now
Were in there now
what I don't have a lot it is voice! hahaha
Cant open the meeting link, Chris is still in the meeting with Exensio I think
Yeaah!
Got it back
Cristobal lost power, christopher are you around?
To all! Alfred weekly meeting in 20 minutes
Just whatsapp me and I'll hop on.
Hi, I can attend alfred meeting if you need an extra hand.
Alfred meeting at 10am ET
Nice notes!
Assets - entermedia as an index, to find our assets,
Assets we difficult to parce, index and find the assets quickly.
Move of file folder, only work in entermedia, not possible anymore. just 1 person responsible for IT.
1 crash mac servers - 2 volumes crashes, no local backup.
No recovery for full server.
On premise server is coming!
Manually clean up files after that.
Once cleaned up: much better position to work better with entermedia, want to work and improve working with eMedia Workspace.
They have metadata for all products, database
Want to tie metadata + products in PIM system
Full time project, couple of months of clean up data.
What are going to do:
- Every folder with a status: in process, pending, completed to check of entire folder hierarchy.
Free standing file, itemnumber that doesnt belong to a folder, how to include them?
They are using a corporate account dropbox
- migrate everything from dropbox too entermedia eventually
Make project plan
dates
deliverables
- Upgrade existing site to this version
- see if that fix downloading / re import everything / we need a clean import.
- Open files and folders (warn for big folders 1gb) size limit. Progress indicator.
- Make folder hierarchy
GOAL: PRIO view all files on eMedia through download or open button.
Recommend:Add existing dropbox files and sync them up as well.
meeting soon for:
better workspace tool
Set up weekly meetings with another colleque
Waiting for IT group now, then we team up for the project. Get them workable in eMedia.
Friday calender - meeting next week.
Automating
Support your team
- meeting alfred standard weekly.
Two volumes crashed, no local backup.
Problem
Keep losing files,
Way too many files.
Limited budget
Solution
Using Entermedia,
Organizing files with metadata and AI
Building a friendly and powerful research tool
And Centralized
Vision
Migrating to a faster server and migrating archives to emedia with dropbox integration, then fine-tune search features.
Effectively organizing and storing to search hundreds of thousands of publications in total (300,000)
The goal is to be able to select all of the assets associated and then import them Into the project and say it’s signed off on, that we found all of the files for the next person to come in. But this never came to fruition due to employee turnover and layoffs. He’d like to continue with this vision
Important part: That we’re not missing out on individual files outside of folders and top levels.
Tasks:
Do more frequent scans.
Optimization and how eMedia can help:
* Check to make sure thumbnail conversion is complete. Making rescans more faster.
* Create an Automation tool for client.
* Working with 2023 version, adding features for uploading and downloading from your desktop version.
* Make it so you can see all folders related to each file, adding descriptions, metadata etc.
* Start building relationships using tags,
* We put together a B site to view. With organizational structures
* More user friendly approach to archives.
* The idea is so you can search from multiple places, by EG Product number, or other defining metadata.
* Attaching 1 to many production folders, attached to different places currently.
Where they’re at and how they’re going to move forward:
They’ve been working with Dropbox after the pandemic hit. IT department discovered that the VPN was terrible and grabbing files from server was absolutely terrible. No possibility of uploading updated assets to our server. New ITI doesn’t understand permissions.
1. Strategy is to have a list of specific products they want to clean up to start.
2. First steps is once IT has a file server up and running and migrated assets to server, That’ll be a good opportunity to start over and think about starting with a new index.
3. Focus on priorities first, then eventually clean up everything.
4. Start with a Reindex (R sync)
5. Migrate from Dropbox to Entermedia using integration
6. Include metadata:
7. Creation, modification, file name, size, location, type
8. Utilize Entermedia to clean up the assets. Will hire a team of 3 employees.
9. Have a thumbnail with supporting metadata for each asset
Challenges they’ve faced:
They’ve been backing up to amazon glacier, but had a huge problem with full server recovery after a major failure. They throttle download speeds so directories that were corrupted have been recovering since December of last year.
Highlighted pain points are mainly with file server upload/download speeds, VPN and need a better server.
Need better archives, to be in a better positioned to move towards intermedia for DAM and PIM solution
Quick updates on eMedia Side:
Working with a lot of clients in a similar situation,
Building a centralized standardized library is a common theme
We have a new version in development
And running it for a few large clients.
Travis’s insights:
1. Problem is assets are not linking on their current PIM system which is a problem.
When you search a product number:
1. Top level folder with product number then returns all individual files outside of the folders with that product number.
2. The search results only return top level directories, which helps.
3. But underneath he has all the files that live outside of the folders. The question is do those files want to be gathered? How do you narrow things down?
4. A tricky part of us, is that we don’t just do the books, we also have performance sets, which is their own animal (EG a concert band) Roughly 20-30 individual parts. But they have another record that represents that publication, but they also sell the score by itself. CB18361- But most assets they only sell one version. But it would be nice to relate and indicate other pub items that are related. EG single scores, or group scores to a concert band performance series.
5. There’s a lot of metadata that is useful to have eg: these files are from the 2023 concert band release, would be very useful. Question EG “Hey can you send me all of 2022s choral products?” Or “all of the harp music you have?” Or “all of the quartet music you have by Bach?” Or “all of the music you have by shoernburg?” Filter by tags. Add columns.
6.
Important part: Not missing out on individual files outside of folders and top levels. They want to utilize all the useful data they have.
I resend the google invites, but you need to accept it.
Alfrd prep meeting today at 10am CST / meeting is at 10.30 am CST
>https://zoom.us/j/97692876614?pwd=RWFISHFSTTNQbHRHZVE0RUYyZVZSQT09
>https://zoom.us/j/97692876614?pwd=RWFISHFSTTNQbHRHZVE0RUYyZVZSQT09
I'm testing locally with a slice of 10 rows in my computer
File Download failed couple times yesterday, I ran it in an Screen so it finished that way
Hello, Im testing the Lookups, looks like they are not linking the other tables
How did you do?
import org.entermediadb.asset.importer.BaseImporter
import org.entermediadb.asset.util.Row
import org.openedit.Data
class CsvImporter extends BaseImporter
{
/**
* This is an example of making a field lower case
*/
protected void addProperties( Row inRow, Data inData)
{
super.addProperties( inRow, inData);
//createLookUp(inSearcher.getCatalogId(),inData,"Division","val_divisions");
}
}
CsvImporter csvimporter = new CsvImporter()
{
protected void createMetadata(Header inHeader)
{
List newnames = new ArrayList();
for (Iterator iterator = inHeader.getHeaderNames().iterator(); iterator.hasNext();)
{
String header = (String) iterator.next();
if(header.contains(".")){ //TODO: What about english and spanish language imports?
continue;
}
if( header == "height")
{
header = "pubheight";
}
newnames.add(header);
}
inHeader.getHeaderNames().clear();
inHeader.setHeaders(newnames);
super.createMetadata(inHeader);
}
};
csvimporter.setSeparator("|".charAt(0));
csvimporter.setModuleManager(moduleManager);
csvimporter.setContext(context);
csvimporter.addDbLookUp("genre");
csvimporter.setLog(log);
csvimporter.setMakeId(false);
csvimporter.importData();
Lookup:
Instrument
Format
Genre
Level
Description -> longcaption?
Pub_Date -> Entity_Date
Site Title -> Name
Series can be a Entity
width and height renamed
pubitem -> PrimaryImage
Create Product Folder entity that is related to PubProduct
Add a way to set uploadsourcepath with Category searcher/browser
Setup hotfolder on new B site with no metadata. To re-create assets and category tree. Use all old thumbnails
I set up alfredB site (m37 was strange turned into messed up DNS) , also installation was not ok at the first try, but now it is working. Im trying to connect to ftp to see the list but not sure if we require credentials
Any progress on this friday? If no we have to prepare this before the meeting on wednesday.
Hi Cristobal! That should be ftp1.alfred.com signing in with guest access. We have a few data files here, but the full inventory one is in this location:ftp1.alfred.com/CatData/DailyAll/ALF_ALL_ALL.txt
Meeting planned demo eMedia Workspace
Set up B-site for travis to show demo this week 23-27 october
