- 3008 Views
- 2 replies
- 1 kudos
Delta tables and YOLO computer vision tasks
Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...
- 3008 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Andrewcon and @jnap , Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is ...
- 1 kudos
- 1231 Views
- 0 replies
- 4 kudos
Best practices for writing efficient SQL queries in Databricks
Are your SQL queries running slow or consuming excessive resources? As data grows in size and complexity, writing optimized SQL queries becomes more crucial than ever. Knowing the right techniques can make all the difference.In my latest blog, I expl...
- 1231 Views
- 0 replies
- 4 kudos
- 544 Views
- 0 replies
- 1 kudos
🚀 Simplifying Complex Data Pipelines with Databricks DLT
Are you dealing with complex XML inputs and wondering how to process them efficiently in a scalable data platform? Look no further! Delta Live Tables (DLT) in Databricks revolutionize how we handle complex data formats like XML. With its declarative...
- 544 Views
- 0 replies
- 1 kudos
- 25584 Views
- 6 replies
- 3 kudos
Resolved! Invalid Characters in Column Names " ,;{}()\n\t="
I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...
- 25584 Views
- 6 replies
- 3 kudos
- 60707 Views
- 4 replies
- 5 kudos
Best practices around bronze/silver/gold (medallion model) data lake classification?
What's the best way to organize our data lake and delta setup? We’re trying to use the bronze, silver and gold classification strategy. The main question is how do we know what classification the data is inside Databricks if there’s no actual physica...
- 60707 Views
- 4 replies
- 5 kudos
- 3686 Views
- 6 replies
- 1 kudos
Presto hive table to delta table conversion
Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with ex...
- 3686 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @SQL @jose_gonzalez , Have you tried code conversion tool fromTravinto technologies ? They have hive to delta table conversion
- 1 kudos
- 2663 Views
- 2 replies
- 0 kudos
Need guidance on connecting to Azure Databricks using JDBC Protocol
Step 1: Download and Reference the JDBC Driver Download the Databricks JDBC Driver: Visit the Databricks JDBC Driver download page. Download the appropriate version for your operating system. Extract the DatabricksJDBC42.jar file from the downloaded ...
- 2663 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @subhankar , Good Day! Looking at the error you are getting here shows that it tries to find some kind of JVM file and probably refers to the JAVA_HOME variable to achieve it. It looks as if it is not set correctly in your Environment Variables. ...
- 0 kudos
- 2708 Views
- 3 replies
- 1 kudos
How can I get logging or print output from a Delta Live Table workflow.
I'm trying to debug a task that is a DLT workflow and I've tried putting in log statements and print statements but I can't seem to see the output in the event log after the run nor can I see the print statements anywhere. Can someone point me to whe...
- 2708 Views
- 3 replies
- 1 kudos
- 1 kudos
Refer to this answer https://community.databricks.com/t5/data-engineering/how-to-print-out-logs-during-dlt-pipeline-run/td-p/82303
- 1 kudos
- 2790 Views
- 1 replies
- 0 kudos
Email Extraction
Hi , Hope you are doing well. I was trying to extract a specific email attachment from the outlook, and inject into the dbfs loaction, but something went wrong. Could you please help. I am hereby giving the code whcih I used. import imaplibimport em...
- 2790 Views
- 1 replies
- 0 kudos
- 0 kudos
If you face issues with IMAP, consider using Microsoft Graph API for email access. It provides robust support for Outlook without handling IMAP details and enhances security with OAuth2 tokens.Followed is a sample script, but I didn't tested it: pip ...
- 0 kudos
- 6733 Views
- 2 replies
- 2 kudos
Resolved! Error API calling with Service Principal Secret
Hi,I am working on Databricks workspace setup on AWS and trying to use Service Principal to execute API calls (CI/CD) deployment through Bitbucket. So I created secret for the service principal and trying to test the token. The test failed with below...
- 6733 Views
- 2 replies
- 2 kudos
- 2 kudos
I have been able to resolve this issue. Apparently you need to generate access token using service principal client id and client secret. saurabh18cs solution is more relevant to Azure Databricks. Got below link from Databricks which provide generic...
- 2 kudos
- 3361 Views
- 1 replies
- 0 kudos
Resolved! Databricks Fundamentals Certificate
Dear Team,I have successfully completed the Databricks Fundamentals training and aced the certificate quiz with a perfect score of 200 out of 200. However, I have not yet received the certificate. Can you please let me know the expected timeline for ...
- 3361 Views
- 1 replies
- 0 kudos
- 0 kudos
You should be able to receive it immediately. Can you share your use id with which you have taken the quiz?
- 0 kudos
- 11953 Views
- 12 replies
- 3 kudos
OOM while loading a lot of data through JDBC
public void bigDataTest() throws Exception { int rowsCount = 100_000; int colSize = 1024; int colCount = 12; String colValue = "'"+"x".repeat(colSize)+"'"; String query = "select explode(s...
- 11953 Views
- 12 replies
- 3 kudos
- 2822 Views
- 5 replies
- 2 kudos
Resolved! Merge operation stuck on scanning files for matches
HiI'm executing simple merge, however it always stucks at "MERGE operation - scanning files for matches". Both delta tables are not big - source has about 100MiB in 1 file and target has 1,5GiB, 7 files, so it should be quite fast operation, however ...
- 2822 Views
- 5 replies
- 2 kudos
- 2 kudos
Well, in the end, it was caused by skewed data. Document_ID was -1 for returns in sales, so a big part of the table was filled with -1 values. Adding an extra column to the merger solved the problem.This article helped me a lot: https://www.databrick...
- 2 kudos
- 3824 Views
- 1 replies
- 1 kudos
Resolved! AI-Suggested Comments Not Appearing for Delta Live Tables Populated Tables
I'm working with Delta Live Tables (DLT) in Databricks and have noticed that AI-suggested comments for columns are not showing up for tables populated using DLT. Interestingly, this feature works fine for tables that are not populated using DLT. Is t...
- 3824 Views
- 1 replies
- 1 kudos
- 1 kudos
It's because materialized view in DLT (MV) and streaming table in DLT (ST) don't support ALTER (which is needed to persist those AI generated comments)
- 1 kudos
- 2809 Views
- 3 replies
- 1 kudos
Resolved! Change spark configs in Serverless compute clusters
Howdy!I wanted to know how I can change some spark configs in a Serverless compute. I have a base.yml file and tried placing: spark_conf: - spark.driver.maxResultSize: "16g"but I still get his error:[CONFIG_NOT_AVAILABLE] Configuration spark.driv...
- 2809 Views
- 3 replies
- 1 kudos
- 1 kudos
To address the memory issue in your Serverless compute environment, you can consider the following strategies: Optimize the Query: Filter Early: Ensure that you are filtering the data as early as possible in your query to reduce the amount of data b...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 786 | |
| 369 | |
| 349 | |
| 287 |
