- 3573 Views
- 2 replies
- 2 kudos
Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks
I testing Spark Streaming working withSASL_SSL enabled kafka broker in a notebook.as per this guide https://docs.databricks.com/spark/latest/structured-streaming/kafka.htmli have copied jsk files in an s3 bucket and mounted it in dbfs.In notebook wh...
- 3573 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks..Yes '/dbfs/mnt/xxxx/kafka.client.truststore.imported.jks' path worked. Also other workaround we got it working, is copy the file from s3 to filesystem using init script and use filepath.
- 2 kudos
- 6758 Views
- 4 replies
- 1 kudos
Inconsistent duplicated row with Spark (Databricks on MS Azure)
I'm having a weird behavior with Apache Spark, which I run in a Python Notebook on Azure Databricks. I have a dataframe with some data, with 2 columns of interest: name and ftimeI found that I sometime have duplicated values, sometime not, depending ...
- 6758 Views
- 4 replies
- 1 kudos
- 1 kudos
I would like to see how you create the df dataframe.In pyspark you can get weird results if you do not clear state, or when you reuse dataframe names.
- 1 kudos
- 4249 Views
- 3 replies
- 2 kudos
Resolved! Cannot use Web Terminal when creating cluster with Custom container.
I follow this guide to create cluster with custom container: https://docs.databricks.com/clusters/custom-containers.htmlHowever, when cluster created, I coudn't access to web terminal. It resulted in 502 bad gateway.
- 4249 Views
- 3 replies
- 2 kudos
- 2 kudos
This is a limitation at the moment. Enabling Docker Container Services disables web terminal.https://docs.databricks.com/clusters/web-terminal.html#limitations
- 2 kudos
- 8245 Views
- 6 replies
- 2 kudos
Resolved! Delta log statistics - timestamp type not working
Hello team!As per the documentation, I understand that the table statistics can be fetched through the delta log (eg min, max, count) in order to not read the underlying data of a delta table.This is the case for numerical types, and timestamp is sup...
- 8245 Views
- 6 replies
- 2 kudos
- 5460 Views
- 3 replies
- 10 kudos
Resolved! Change cluster mid-pipeline
I have a notebook functioning as a pipeline, where multiple notebooks are chained together. The issue I'm facing is that some of the notebooks are spark-optimized, others aren't, and what I want is to use 1 cluster for the former and another for the ...
- 5460 Views
- 3 replies
- 10 kudos
- 10 kudos
Yes, you can achieve this by setting two different job clusters. In the screenshot, you can see I have used 2 job clusters PipelineTest and pipelinetest2. You can refer the doc https://docs.databricks.com/jobs.html#cluster-config-tips
- 10 kudos
- 6730 Views
- 4 replies
- 4 kudos
Calling Databricks API from Databricks notebook
A similar question has already been added, but the reply is very confusing to me. Basically, for automated jobs, I want to log the following information from inside a Python notebook that runs in the job: - What is the cluster configuration (most im...
- 6730 Views
- 4 replies
- 4 kudos
- 4 kudos
hi @Thomas Kastl​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 4 kudos
- 2847 Views
- 3 replies
- 2 kudos
best practice of using data bricks API
Hello, I'm building a Databricks connector to allow users to issue command/SQL from a web app.In general, I think the REST API is okay to work with, though it's pretty tedious to write wrap code for each API call.[Q1]Is there an official (or semi-off...
- 2847 Views
- 3 replies
- 2 kudos
- 2 kudos
I don't know if I fully understand DBX, sounds like a job client to manage jobs and deployment and I don't see NodeJS support for this project yet. My question was about how to "stream" query results back from Databricks in a NodeJs application, curr...
- 2 kudos
- 1934 Views
- 2 replies
- 0 kudos
How to write to a folder in a Azure Data Lake container using Delta?
Hi All,How to write to a folder in a Azure Data Lake container using Delta?When I run:write_mode = 'overwrite' write_format = 'delta' save_path = '/mnt/container-name/folder-name' df.write \ .mode(write_mode) \ .format(write_format) \ ....
- 1934 Views
- 2 replies
- 0 kudos
- 1742 Views
- 1 replies
- 0 kudos
How to explain the huge time latency between two jobs? How to optimize the job to reduce the latency ?
I have met a problem , you can see in the picture as followed: there is some long delay between some jobs , I don't understand what happened and how to optimize the job ? Can anybody help me ? Thanks a lot.
- 1742 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jieping zhang​,Did you check the driver's logs? do you see any error messages? please provide more details
- 0 kudos
- 6730 Views
- 8 replies
- 4 kudos
Unable to read an XML file of 9 GB
Hello,We have a large XML file (9 GB) that we can't read.We have this error : VM size limitBut how can we change the VM size limit ?We have tested many clusters, but no one can read this file.Thank you for your help.
- 6730 Views
- 8 replies
- 4 kudos
- 4 kudos
Hi @Salah K.​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.
- 4 kudos
- 2114 Views
- 2 replies
- 3 kudos
Is the promise of a data lake simply about data science, data analytics and data quality or can it also be an integral part of core transaction processing also?
Upfront, I want to let you know that I'm not a veteran data jockey, so I apologize if this topic has been covered already or is simply too basic or narrow for this community. That said, I do need help so please feel free to point me in another direc...
- 2114 Views
- 2 replies
- 3 kudos
- 3 kudos
@Marc Barnett​ , Databricks’ Lakehouse architecture is the ideal data architecture for data-driven organizations. It combines the best qualities of data warehouses and data lakes to provide a single solution for all major data workloads and supports ...
- 3 kudos
- 7566 Views
- 7 replies
- 4 kudos
Pulling list of running jobs using JOBS API 2.1
I need to find out all jobs which are currently running and not get other jobsThe below command fetches all the jobscurl --location --request GET 'https://xxxxxx.gcp.databricks.com/api/2.1/jobs/list?active_only=true&expand_tasks=true&run_type=JOB_RUN...
- 7566 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @Sumit Rohatgi​ It seems like active_only=true only applies to jobs/runs/list API and not to jobs/list.Can you please try the jobs/runs/list API?
- 4 kudos
- 5638 Views
- 5 replies
- 4 kudos
Resolved! Databricks notebook command logging
Hello Community,I am trying to search for Databricks notebook command logging feature for compliance purpose.My requirement is to log the exact spark sql fired by user.I didnt get spark sql (notebook command) tracked under this azure diagnostic logs....
- 5638 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @C P​ we don't have this feature implemented, however, there is already an existing idea available in our idea portal here: https://databricks.aha.io/features/DB-7583.You can check and vote the same.
- 4 kudos
- 1741 Views
- 1 replies
- 0 kudos
Real Time data processing
Say I am getting a customer record from an website. I want to read the massage & then insert/update that one to snowflake table , depending on the records insert/update is successful I need to respond back the success / failure massage in say 1 sec. ...
- 1741 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @CHANDAN NANDY​ Just checking in with you.Does @Kaniz Fatma​'s answer help? If it does, would you be happy to mark it as best? If it doesn't, please tell us so we can help you further.Thanks!
- 0 kudos
- 3664 Views
- 1 replies
- 1 kudos
Are there implications moving Managed Table, and mounting as External.
The scenario is "A substaincial amount of data needs to be moved from a legacy Databricks that has Managed Tables, to a new E2 Databrick. The new bucket will be a dedicated Datalake rather than the Workspace Bucket so they will be External Tables."U...
- 3664 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey there @John Brandborg​ Hope everything is going great! Just wanted to check in if you were able to resolve your issue would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to ...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
