- 10942 Views
- 4 replies
- 2 kudos
Resolved! Read JSON with backslash.
Hello guys.I'm trying to read JSON file which contains backslash and failed to read it via pyspark.Tried a lot of options but didn't solve this yet, I thought to read all the JSON as text and replace all "\" with "/" but pyspark fail to read it as te...
- 10942 Views
- 4 replies
- 2 kudos
- 2 kudos
@orian hindi​ - Would you be happy to post the solution you came up with and then mark it as best? That will help other members.
- 2 kudos
- 8889 Views
- 8 replies
- 0 kudos
Hi All, I am trying to read a csv file from datalake and loading data into sql table using Copyinto. am facing an issue   Here i created one table wit...
Hi All,I am trying to read a csv file from datalake and loading data into sql table using Copyinto.am facing an issue Here i created one table with 6 columns same as data in csv file.but unable to load the data.can anyone helpme on this
- 8889 Views
- 8 replies
- 0 kudos
- 0 kudos
Thanks Werners for your Reply,How to pass schema(ColumnName && Types) to CSV file ??
- 0 kudos
- 9381 Views
- 4 replies
- 3 kudos
Resolved! Pyspark logging - custom to Azure blob mount directory
I'm using the logging module to log the events from the job, but it seems the log is creating the file with only 1 lines. The consecutive log events are not being recorded. Is there any reference for custom logging in Databricks.
- 9381 Views
- 4 replies
- 3 kudos
- 3 kudos
@karthick J​ - If Jose's answer helped solve the issue, would you be happy to mark their answer as best so that others can find the solution more easily?
- 3 kudos
- 1367 Views
- 0 replies
- 0 kudos
SQL server error
What is SQL Server 80004005 error? How to fix?
- 1367 Views
- 0 replies
- 0 kudos
- 5532 Views
- 1 replies
- 2 kudos
Resolved! Notebook failing in job-cluster but runs fine in all-purpose-cluster with the same configuration
I have a notebook with many join and few persist operations (which runs fine on all-purpose-cluster (with worker nodes - i3.xlarge and autoscale enabled), but the same notebook failing in job-cluster with the same cluster definition (to be frank the ...
- 5532 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Ajay Nanjundappa​ ,Check "Event log" tab. Search for any spot terminations events. It seems like all your nodes are spot instances. The error "FetchFailedException" is associated with spot termination nodes.
- 2 kudos
- 4594 Views
- 2 replies
- 3 kudos
Resolved! yarn.nodemanager.resource.memory-mb parameter update
I am currently working on determining proper cluster size for my Spark application and I have a question regarding Hadoop configuration parameter yarn.nodemanager.resource.memory-mb. From what I see, this parameter is responsible for setting the phys...
- 4594 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Andriy Shevchenko​ ,Databricks does not use Yarn. I recommend you to try to use Databricks community edition link to get familiar and explore. You can check Ganglia UI to see how is the cluster utilization, memory, cpu, IO, etc
- 3 kudos
- 9659 Views
- 3 replies
- 1 kudos
How to access databricks secret in global ini file
How to access databricks secret in global ini file. {{secrets/scope/key}} doesnt work. Do i have to put that inside quotes
- 9659 Views
- 3 replies
- 1 kudos
- 1 kudos
hi @SEBIN THOMAS​ ,I would like to share the docs here are you getting any error messages? like @Hubert Dudek​ mentioned, please share more details and error message in case you are getting any.
- 1 kudos
- 3948 Views
- 5 replies
- 2 kudos
Which rest API to use in order to list the groups that belong to a specific user
Which rest API to use in order to list the groups that belong to a specific user
- 3948 Views
- 5 replies
- 2 kudos
- 2 kudos
@Mohit Miglani​ ,Make sure to select the best option so the post will be moved to the top and will help in case more users have this question in the future.
- 2 kudos
- 4488 Views
- 3 replies
- 4 kudos
Resolved! adding databricks to my application
I am developing an application. I want to use databricks in my application. I developed that with python and godot. how can I have data bricks in my application?
- 4488 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Ensiyeh Shojaei​ ,Which cloud service are you using? According to the cloud provider, you will have a list of tools that can help you to connect and interact in your application.
- 4 kudos
- 4563 Views
- 1 replies
- 1 kudos
Resolved! Downstream duration timeout
I'm trying to upload a file that is .5GB for a school lab and when I drag the file to DBFS it uploads for about 30 seconds and then I receive a downstream duration timeout error. What can I do to solve this issue?
- 4563 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Jason Schmit​ ,Your file might be too large to upload by using the upload interface docs I will recommend to split it up into smaller files. You can also use DBFS CLI, dbutils to upload your file.
- 1 kudos
- 13986 Views
- 7 replies
- 3 kudos
How to avoid DataBricks Secret Scope from exposing the value of the key resides in Azure Key Vault?
I have created a key in Azure Key Vault to store my secrets in it. In order to use it securely in Azure DataBricks, have created the secret scope and configured the Azure Key Vault properties. Out of curiosity, just wanted to check whether my key is ...
- 13986 Views
- 7 replies
- 3 kudos
- 3 kudos
@Kaniz Fatma​ is any fix coming soon for this? this is a big security loophole The docs say that "To ensure proper control of secrets you should use Workspace object access control (limiting permission to run commands) " --- if i prevent access to ru...
- 3 kudos
- 5349 Views
- 3 replies
- 2 kudos
Resolved! Is it possible to disable retryWrites using .option()?
Hello everyone,I'm trying to write to DocumentDB using org.mongodb.spark:mongo-spark-connector_2.12:3.0.1. The DocDB is version 4 which doesn't support Retryable Writes so I disabled the feature setting option "retryWrites" to "false" (also tried wit...
- 5349 Views
- 3 replies
- 2 kudos
- 2 kudos
@Hugh Vo​ - If Sajehs's answer resolved the issue, would you be happy to mark their answer as best?
- 2 kudos
- 2968 Views
- 2 replies
- 2 kudos
Resolved! Are notebooks encrypted even if no CMK is provided?
This document (https://docs.databricks.com/security/keys/customer-managed-keys-managed-services-aws.html) describes how to use a customer managed key to encrypt notebooks in the control plane.We would please like to verify: if no CMK is provided, are...
- 2968 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @Nathan Buesgens​ , from a high level point of view, by default, notebooks source code and metadata in the control plane are encrypted at rest in AWS RDS using AWS KMS with a Databricks-managed Key. But there is other data related to notebooks ...
- 2 kudos
- 4588 Views
- 3 replies
- 3 kudos
Resolved! How do I create a job for a notebook not in the /Users/ directory?
I am setting up a job to to load data from S3 into Delta using Auto loader. I can do this fine in interactive mode. When trying to create a job in the UI. I can select the notebook in the root directory I created for the project within the create jo...
- 4588 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Ken Pendergast​, you are supposed to be able to reference any Notebook you have the right permissions on so it looks like you are running into a bug, can you please reach out to support or email me directly with your workspace ID? My email is jan...
- 3 kudos
- 3055 Views
- 3 replies
- 0 kudos
Resolved! SQL Issues with 10.0 runtime
I was testing my sqls with new 10.0 runtime and found some interesting/weird thing. The same sql with explode function fails for some scenarios in 10.0! Could not figure out yet the reason
- 3055 Views
- 3 replies
- 0 kudos
- 0 kudos
@Yadhuram MC​ if the issue persists, please email me at bilal dot aslam at databricks dot com. I would like to get to the root of this issue. It
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 561 | |
| 349 | |
| 287 |
