- 499 Views
- 1 replies
- 0 kudos
Missing Rows When Reading Data from Impala Kudu to Databricks Using JDBC
Hi everyone,I’m working on a data ingestion process where I need to read data from an Impala Kudu table into Databricks using the JDBC connector. However, I’m experiencing an issue where some rows are missing in the data read. For instance, if there ...
- 499 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kuke Have you checked whether the partitioning is configured correctly?If disabling partitioning (creating a single partition) allows you to retrieve 100,000 rows, but enabling partitioning results in only 99,000 rows, it is likely that the partitio...
- 0 kudos
- 2756 Views
- 5 replies
- 0 kudos
Resolved! How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?
I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need ...
- 2756 Views
- 5 replies
- 0 kudos
- 0 kudos
sure, happy to assist, if any of my responses was able to help you would really appreciate if you can accept it as a solution
- 0 kudos
- 3258 Views
- 2 replies
- 1 kudos
Delta tables and YOLO computer vision tasks
Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...
- 3258 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Andrewcon and @jnap , Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is ...
- 1 kudos
- 1583 Views
- 0 replies
- 4 kudos
Best practices for writing efficient SQL queries in Databricks
Are your SQL queries running slow or consuming excessive resources? As data grows in size and complexity, writing optimized SQL queries becomes more crucial than ever. Knowing the right techniques can make all the difference.In my latest blog, I expl...
- 1583 Views
- 0 replies
- 4 kudos
- 683 Views
- 0 replies
- 1 kudos
🚀 Simplifying Complex Data Pipelines with Databricks DLT
Are you dealing with complex XML inputs and wondering how to process them efficiently in a scalable data platform? Look no further! Delta Live Tables (DLT) in Databricks revolutionize how we handle complex data formats like XML. With its declarative...
- 683 Views
- 0 replies
- 1 kudos
- 27447 Views
- 6 replies
- 3 kudos
Resolved! Invalid Characters in Column Names " ,;{}()\n\t="
I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...
- 27447 Views
- 6 replies
- 3 kudos
- 62440 Views
- 4 replies
- 5 kudos
Best practices around bronze/silver/gold (medallion model) data lake classification?
What's the best way to organize our data lake and delta setup? We’re trying to use the bronze, silver and gold classification strategy. The main question is how do we know what classification the data is inside Databricks if there’s no actual physica...
- 62440 Views
- 4 replies
- 5 kudos
- 3981 Views
- 6 replies
- 1 kudos
Presto hive table to delta table conversion
Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with ex...
- 3981 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @SQL @jose_gonzalez , Have you tried code conversion tool fromTravinto technologies ? They have hive to delta table conversion
- 1 kudos
- 3186 Views
- 2 replies
- 0 kudos
Need guidance on connecting to Azure Databricks using JDBC Protocol
Step 1: Download and Reference the JDBC Driver Download the Databricks JDBC Driver: Visit the Databricks JDBC Driver download page. Download the appropriate version for your operating system. Extract the DatabricksJDBC42.jar file from the downloaded ...
- 3186 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @subhankar , Good Day! Looking at the error you are getting here shows that it tries to find some kind of JVM file and probably refers to the JAVA_HOME variable to achieve it. It looks as if it is not set correctly in your Environment Variables. ...
- 0 kudos
- 3286 Views
- 3 replies
- 1 kudos
How can I get logging or print output from a Delta Live Table workflow.
I'm trying to debug a task that is a DLT workflow and I've tried putting in log statements and print statements but I can't seem to see the output in the event log after the run nor can I see the print statements anywhere. Can someone point me to whe...
- 3286 Views
- 3 replies
- 1 kudos
- 1 kudos
Refer to this answer https://community.databricks.com/t5/data-engineering/how-to-print-out-logs-during-dlt-pipeline-run/td-p/82303
- 1 kudos
- 4384 Views
- 1 replies
- 0 kudos
Email Extraction
Hi , Hope you are doing well. I was trying to extract a specific email attachment from the outlook, and inject into the dbfs loaction, but something went wrong. Could you please help. I am hereby giving the code whcih I used. import imaplibimport em...
- 4384 Views
- 1 replies
- 0 kudos
- 0 kudos
If you face issues with IMAP, consider using Microsoft Graph API for email access. It provides robust support for Outlook without handling IMAP details and enhances security with OAuth2 tokens.Followed is a sample script, but I didn't tested it: pip ...
- 0 kudos
- 8070 Views
- 2 replies
- 2 kudos
Resolved! Error API calling with Service Principal Secret
Hi,I am working on Databricks workspace setup on AWS and trying to use Service Principal to execute API calls (CI/CD) deployment through Bitbucket. So I created secret for the service principal and trying to test the token. The test failed with below...
- 8070 Views
- 2 replies
- 2 kudos
- 2 kudos
I have been able to resolve this issue. Apparently you need to generate access token using service principal client id and client secret. saurabh18cs solution is more relevant to Azure Databricks. Got below link from Databricks which provide generic...
- 2 kudos
- 4098 Views
- 1 replies
- 0 kudos
Resolved! Databricks Fundamentals Certificate
Dear Team,I have successfully completed the Databricks Fundamentals training and aced the certificate quiz with a perfect score of 200 out of 200. However, I have not yet received the certificate. Can you please let me know the expected timeline for ...
- 4098 Views
- 1 replies
- 0 kudos
- 0 kudos
You should be able to receive it immediately. Can you share your use id with which you have taken the quiz?
- 0 kudos
- 12462 Views
- 12 replies
- 3 kudos
OOM while loading a lot of data through JDBC
public void bigDataTest() throws Exception { int rowsCount = 100_000; int colSize = 1024; int colCount = 12; String colValue = "'"+"x".repeat(colSize)+"'"; String query = "select explode(s...
- 12462 Views
- 12 replies
- 3 kudos
- 3219 Views
- 5 replies
- 2 kudos
Resolved! Merge operation stuck on scanning files for matches
HiI'm executing simple merge, however it always stucks at "MERGE operation - scanning files for matches". Both delta tables are not big - source has about 100MiB in 1 file and target has 1,5GiB, 7 files, so it should be quite fast operation, however ...
- 3219 Views
- 5 replies
- 2 kudos
- 2 kudos
Well, in the end, it was caused by skewed data. Document_ID was -1 for returns in sales, so a big part of the table was filled with -1 values. Adding an extra column to the merger solved the problem.This article helped me a lot: https://www.databrick...
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 491 | |
| 349 | |
| 287 |
