cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Erik
by Valued Contributor II
  • 3879 Views
  • 4 replies
  • 4 kudos

Liquid clustering with structured streaming pyspark

I would like to try out liquid clustering, but all the examples I see seem to be SQL tables created from selecting from other tables. Our gold tables are pyspark tables written directly to a table, e.g. like this: silver_df.writeStream.partitionBy(["...

  • 3879 Views
  • 4 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

I did not find anything in the docs either.  I suppose a pyspark version will come in the future?

  • 4 kudos
3 More Replies
Yoshe1101
by New Contributor III
  • 2188 Views
  • 2 replies
  • 1 kudos

Resolved! Cluster terminated. Reason: Npip Tunnel Setup Failure

Hi, I have recently deployed a new Workspace in AWS and getting the following error when trying to start the cluster:"NPIP tunnel setup failure during launch. Please try again later and contact Databricks if the problem persists. Instance bootstrap f...

  • 2188 Views
  • 2 replies
  • 1 kudos
Latest Reply
Yoshe1101
New Contributor III
  • 1 kudos

Finally, this error was fixed by changing the DHCP configuration of the VPC.

  • 1 kudos
1 More Replies
MichaelO
by New Contributor III
  • 2614 Views
  • 4 replies
  • 2 kudos

Resolved! Call python image function in pyspark

I have a function for rotating images written in python:from PIL import Image def rotate_image(image, rotation_angle): im = Image.open(image) out = im.rotate(rotation_angle, expand = True) return outI now want to use this function as a pyspark ...

  • 2614 Views
  • 4 replies
  • 2 kudos
Latest Reply
Raluka
New Contributor III
  • 2 kudos

Stock photos, I've come to realize, are the catalysts of imagination. This website's vast reservoir of images new york seal sparks ideas that ripple through my projects. They empower me to envision the previously unimagined, helping me breathe life i...

  • 2 kudos
3 More Replies
ottomes
by New Contributor II
  • 1750 Views
  • 3 replies
  • 0 kudos

What is my subscription plan?

I am working as data engineer I was about checking the subscription plan. I would like to know how I can check the subscription plan. I am "Admin" but I cannot "Manage accounts" on Databricks workspace portal.This subscription information is pretty i...

Data Engineering
calculation
DBU
Premium
pricing
Standard
  • 1750 Views
  • 3 replies
  • 0 kudos
Latest Reply
ottomes
New Contributor II
  • 0 kudos

Hey, not really. Update you: if you try to add ACL to a secret scope then you will be sure that your subscription is Enterprise or Standard, because either you succeed then you are working with Enterprise or the API respond with the Standard subscrip...

  • 0 kudos
2 More Replies
alonisser
by Contributor
  • 706 Views
  • 2 replies
  • 0 kudos

Trying to vacuum a table that is constantly being "createdOrReplaced"

and it seems that older data (From the "replaced" table) isn't being removed, long after the retention period I'd be glad for clues on how to handle this

  • 706 Views
  • 2 replies
  • 0 kudos
Latest Reply
alonisser
Contributor
  • 0 kudos

Thanks, I know that, but the table history shows 30 days, but the actual data size and number of files and all other indicators , correlate to 170 days. 

  • 0 kudos
1 More Replies
Rishitha
by New Contributor III
  • 6073 Views
  • 3 replies
  • 2 kudos

Resolved! DLT pipeline

Hi all!I have a question about setting a target schema. How to set different targets for 2 different tables in the same delta live table pipeline. We have 2 target schemas in a database Bronze_chema and silver_schema.  The pipeline has a streaming ra...

  • 6073 Views
  • 3 replies
  • 2 kudos
Latest Reply
Rishitha
New Contributor III
  • 2 kudos

Thanks again @btafur Hoping for this feature to release soon!

  • 2 kudos
2 More Replies
Jayanth746
by New Contributor III
  • 11083 Views
  • 10 replies
  • 4 kudos

Kafka unable to read client.keystore.jks.

Below is the error we have received when trying to read the stream Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jksCaused by: java.nio.file.NoSuchFileException: /dbfs...

  • 11083 Views
  • 10 replies
  • 4 kudos
Latest Reply
mwoods
New Contributor III
  • 4 kudos

Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss...

  • 4 kudos
9 More Replies
mwoods
by New Contributor III
  • 4683 Views
  • 3 replies
  • 2 kudos

Resolved! Spark readStream kafka.ssl.keystore.location abfss path

Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/23301 - the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azure-databricks...

  • 4683 Views
  • 3 replies
  • 2 kudos
Latest Reply
mwoods
New Contributor III
  • 2 kudos

@Kaniz_Fatma- quick update - managed to find the cause. It's neither of the above, it's a bug in the DataBricks 14.0 runtime. I had switched back to the 13.3 LTS runtime, and that is what caused the error to disappear.As soon as I try to read directl...

  • 2 kudos
2 More Replies
Sahha_Krishna
by New Contributor
  • 779 Views
  • 1 replies
  • 0 kudos

Unable to start Cluster in Databricks because of `BOOTSTRAP_TIMEOUT`

Unable to start the Cluster in AWS-hosted Databricks because of the below reason{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-0634ee9c2d420edc8), status: INSTANCE_INITIALIZIN...

  • 779 Views
  • 1 replies
  • 0 kudos
Latest Reply
Harrison_S
New Contributor III
  • 0 kudos

Hi Sahha, It may be a DNS issue if that wasn't rolled back, can you check the documentation on troubleshooting guide and see if these configurations were rolled back as well? https://docs.databricks.com/en/administration-guide/cloud-configurations/aw...

  • 0 kudos
Vaibhav1000
by New Contributor II
  • 648 Views
  • 1 replies
  • 0 kudos

Spark streaming is not able to assume role

Hello,I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a 403 error.  The code is provided below:spark.readStream .format("s3-sqs") .option("fileFormat", "json") .option("roleArn", roleArn) .option("compressi...

  • 648 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Vaibhav1000, The 403 error you're encountering with the "s3-sqs" format could be due to incorrect configuration or insufficient permissions. You can try the following steps to resolve this issue: 1. Check your IAM role permissions: Ensure that t...

  • 0 kudos
rpaschenko
by New Contributor II
  • 1571 Views
  • 2 replies
  • 2 kudos

Databricks Job issue (run was cancelled bydatabricks and spark UI is not available after 10mins)

Hi!We had an issue on 09/19/2023 - we launched job, run was started, but after 10mins it was cancelled with no reasons. The spark ui is not available (which probably means that claster has not been started at all) and I don’t see any logs even.Could ...

  • 1571 Views
  • 2 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

Was it a one time only error or a recurring one?For the former, I'd check if your vCPU quota was not exceeded, or perhaps there was a temporary issue with the cloud provider,...  Could be a lot of things (lots of moving parts under the hood).For the ...

  • 2 kudos
1 More Replies
Shivani_DB
by New Contributor
  • 601 Views
  • 1 replies
  • 0 kudos

Performance Issues experienced when cluster was upgraded from 10.4 LTS to 11.3 LTS

Performance Issues experienced when cluster was upgraded from 10.4 LTS to 11.3 LTS , The notebooks were running fine with the existing cluster. Soon after the upgrade the notebooks started to fail or exhaust memory executors etc . Any suggestions?

  • 601 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Shivani_DB,  The possible reasons for the issues might be: - Incompatible dependencies with Spark elastic jar - Class not found error: org/sparkproject/guava/cache/CacheLoader Suggestions to resolve the issues: - Check for incompatible dependenci...

  • 0 kudos
kmorton
by New Contributor
  • 956 Views
  • 1 replies
  • 0 kudos

Autoloader start and end date for ingestion

I have been searching for a way to set up backfilling using autoloader with an option to set a "start_date" or "end_date". I am working on ingesting a massive file system but I don't want to ingest everything from the beginning. I have a start date t...

Data Engineering
autoloader
backfill
ETL
ingestion
  • 956 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @kmorton, Databricks Auto Loader does support backfilling to capture any missed files with file notifications. This is achieved by using the cloudFiles.backfillInterval option to schedule regular backfills over your data. However, it does not spec...

  • 0 kudos
robbie1
by New Contributor
  • 1992 Views
  • 4 replies
  • 2 kudos

Resolved! Can't login anymore: Invalid email address or password

Since last Friday i cannot access databricks community any more, which is kinda annoying since my Bachelors dissertation is due in a couple of weeks. I always get the message: "Invalid email address or password Note: Emails/usernames are case-sensiti...

  • 1992 Views
  • 4 replies
  • 2 kudos
Latest Reply
nnaincy
New Contributor III
  • 2 kudos

Hi Team,My community edition databricks cred are locked. I am doing very important project. Please help me resolve the issue Please try that it not gets locked in future as well.Email used for login @Kaniz_Fatma @Sujitha I have sent a email to  commu...

  • 2 kudos
3 More Replies
aicd_de
by New Contributor III
  • 1815 Views
  • 4 replies
  • 2 kudos

Unity Catalog - Writing to PNG Files to Cluster and then using dbutils.fs.cp to send to Azure ADLS2

Hi AllLooking to get some help. We are on Unity Catalog in Azure. We have a requirement to use Python to write out PNG files (several) via Matplotlib and then drop those into an ADLS2 Bucket. With Unity Catalog, we can easily use dbutils.fs.cp or fs....

  • 1815 Views
  • 4 replies
  • 2 kudos
Latest Reply
aicd_de
New Contributor III
  • 2 kudos

Hmm I read something different - someone else had this error because they used a shared cluster - apparently it does not happen on a single user cluster. All those settings are already done and I am a fully admin.

  • 2 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors