- 3296 Views
- 3 replies
- 4 kudos
Issue while reading data from Kafka topic to Spark strutured streaming
py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContextI already disable acl for cluster using "...
- 3296 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Ravi Teja​,Just a friendly follow-up. Do you still need help? if you do, please share more details, like DBR version, standard or High concurrency cluster? etc
- 4 kudos
- 3274 Views
- 2 replies
- 3 kudos
Academy Accreditation - SQL Analyst Associate
Hi Team , So last year I acquired SQL Analyst Associate badge and due for renew this Jan 2023 . However when checked in Databricks Academy couldn't find the course . So has it been retired or removed ? If exists can someone help me with the course d...
- 3274 Views
- 2 replies
- 3 kudos
- 1836 Views
- 3 replies
- 0 kudos
Custom duty charges or any other additional charges on Databricks rewards.
Hi All,Anyone can please confirm i have to pay any custom duty fee or ​any other additional shippment fee for Databricks rewards?​Thanks​
- 1836 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @S S​,Just a friendly follow-up. Do you still need help with this question? please let us know
- 0 kudos
- 8287 Views
- 6 replies
- 0 kudos
Resolved! Databricks Delta Live Table stored as SCD 2 is creating new records when no data changes. How do I stop this?
I have a streaming pipeline that ingests json files from a data lake using autoloader. These files are dumped there periodically. Mostly the files contain duplicate data, but there are occasional changes. I am trying to process these files into a dat...
- 8287 Views
- 6 replies
- 0 kudos
- 0 kudos
For clarity, here is the final code that avoids duplicates, using @Suteja Kanuri​ 's suggestion:import dlt @dlt.table def currStudents_dedup(): df = spark.readStream.format("delta").table("live.currStudents_ingest") return ( df.drop...
- 0 kudos
- 3710 Views
- 5 replies
- 0 kudos
Whenever using the displayHTML method or any python library that requires rendering HTML we get the following error in the results:Â Â Uncaught SyntaxEr...
Whenever using the displayHTML method or any python library that requires rendering HTML we get the following error in the results: Uncaught SyntaxError: Invalid or unexpected tokenWe cannot reproduce this error reliably, and resizing the html window...
- 3710 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi, If you could confirm the whole error stack will help us understanding the issue little clear. Also, please tag @Debayan​ with your next response which will notify me. Thank you!
- 0 kudos
- 4682 Views
- 3 replies
- 0 kudos
FileNotFoundError: [Errno 2] No such file or directory:
I have the following error code in databricks when I want to unzip filesFileNotFoundError: [Errno 2] No such file or directory: but the file is there I already tried several ways and nothing worksI have tried modifying by placing/dbfs/mnt/dbfs/mnt/d...
- 4682 Views
- 3 replies
- 0 kudos
- 0 kudos
@Alfredo Vallejos​ then your file is tar.gz file right, have you tried tar command instead of unzip
- 0 kudos
- 2316 Views
- 2 replies
- 0 kudos
Cloud storage - enabling object versioning?
So i am going to keep this generic as to all cloud provider storage options as its relevant across the board, (GCS, S3 and blob store). Nothing is mentioned in docs as far as i can see. Is there a use case against enabling object versioning in cloud ...
- 2316 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Matt User​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 0 kudos
- 3424 Views
- 4 replies
- 0 kudos
I faced some problem while taking databricks exam
Hi team, Good evening today I got problem while taking the exam my exam is @11:30 but some audio problem it's got reschedule @12:45 again also I faced problem ,question was some time appears and some time it's not so, because this I can't able to ta...
- 3424 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @S Meghala​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...
- 0 kudos
- 9797 Views
- 6 replies
- 3 kudos
Autoloader - understanding missing file after schema update.
Hello,Concerning Autoloader (based on https://docs.databricks.com/ingestion/auto-loader/schema.html), so far what I understand is when it detects a schema update, the stream fails and I have to rerun it to make it works, it's ok.But once I rerun it, ...
- 9797 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi @Lucien Arrio​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 3 kudos
- 4470 Views
- 9 replies
- 3 kudos
Dear Community members, We want to extend our sincere gratitude for attending the Community event - March series on March 31st 2023. Your presence mad...
Dear Community members,We want to extend our sincere gratitude for attending the Community event - March series on March 31st 2023. Your presence made the event a huge success, and we appreciate the time you took to join us. We were thrilled to hear ...
- 4470 Views
- 9 replies
- 3 kudos
- 3 kudos
@Suteja Kanuri​ Hi Suteja. Great initiative. Please plan a common timezone between India and UK/EUR/US so that we can also attend. BTW is there any recorded session that we can go through?
- 3 kudos
- 2221 Views
- 2 replies
- 0 kudos
Delta table acceleration for group by on key columns using ZORDER does not work
What is the best practice for accelerating queries which looks like the following?win = Window.partitionBy('key1','key2').orderBy('timestamp') df.select('timestamp', (F.col('col1') - F.lag('col1').over(win)).alias('col1_diff'))I have tried to use OP...
- 2221 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Hanan Shteingart​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...
- 0 kudos
- 4733 Views
- 3 replies
- 4 kudos
- 4733 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Machireddy Nikitha​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 4 kudos
- 4061 Views
- 1 replies
- 0 kudos
Fatal error: The Python kernel is unresponsive when attempting to query data from AWS Redshift within Jupyter notebook
I am running jupyter notebook on a cluster with configuration: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)Worker type: i3.xlarge 30.5gb memory, 4 coresMin 2 and max 8 workers cursor = conn.cursor() cursor.execute( """ ...
- 4061 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please confirm the usage of your cluster while running this job? you can monitor the performance here: https://docs.databricks.com/clusters/clusters-manage.html#monitor-performance with different metrics. Also, please tag @Debayan​ with...
- 0 kudos
- 10675 Views
- 6 replies
- 1 kudos
Open firewall to Azure Databricks workspace from AWS RDS machine/EC2 machine
Hi All,As part of our solution approach, we need to connect to one of our AWS RDS Oracle databases from Azure Databricks notebook.We need your help to understand which IP range of Azure Databricks to consider to whitelist them on AWS RDS security gro...
- 10675 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Mahesh D​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos
- 14578 Views
- 13 replies
- 9 kudos
dbt workflow job limitations - naming the target? where do docs go?
I'm on unity catalogI'm trying to do a dbt run on a project that works locallybut the databricks dbt workflow task seems to be ignoring the project.yml settings for schemas and catalogs, as well as that defined in the config block of individual model...
- 14578 Views
- 13 replies
- 9 kudos
- 9 kudos
Hi @Jakub K​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest provid...
- 9 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
