- 3211 Views
- 1 replies
- 0 kudos
- 3211 Views
- 1 replies
- 0 kudos
- 0 kudos
@Bhagwan Chaubey​ May be you can give this a try, if this is a Blob Storage account.https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windowsFor Datalake storage, please try belowhttps://do...
- 0 kudos
- 10416 Views
- 4 replies
- 3 kudos
Resolved! Writing Spark data frame to ADLS is taking Huge time when Data Frame is of Text data.
Spark data frame with text data when schema is in Struct type spark is taking too much time to write / save / push data to ADLS or SQL Db or download as csv.
- 10416 Views
- 4 replies
- 3 kudos
- 3 kudos
@shiva Santosh​ Have to checked the count of the dataframe that you are trying to save to ADLS?As @Joseph Kambourakis​ mentioned the explode can result in 1-many rows, better to check data frame count and see if Spark OOMs in the workspace.
- 3 kudos
- 3513 Views
- 1 replies
- 2 kudos
github.com
Do I always need to manually invite members of my AAD tenant to ADB workspace, if I don't have SCIM integration configured?EDIT: solved, it works when you go through Azure Portal and get in with "Launch Workspace" button on the ADB resource overview ...
- 3513 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @pawelmitrus​ For users with Owner or Contributor roles, they should click on the "Launch Workspace" button in the Azure portal. For other users they should be explicitly granted access to the workspace to be able to login.Regards,Arvind
- 2 kudos
- 3395 Views
- 1 replies
- 4 kudos
Resolved! Databricks Community: Cluster Terminated Reason: Unexpected Launch Failure
Hi,I've been encountering the following error when I try to start a cluster, but the status page says everything is fine. Is something happening or are there other steps I can try?Time2022-03-13 14:40:51 EDTMessageCluster terminated.Reason:Unexpected...
- 3395 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Rachel Kelley​ We have some internal service interruptions due to which we had this issue. Our engineering has applied the fix and the cluster startup works as expected. Sincerely apologies for the inconvenience caused here.Regards,Darshan
- 4 kudos
- 8227 Views
- 2 replies
- 0 kudos
How to read a compressed file in spark if the filename does not include the file extension for that compression format?
For example, let's say I have a file called some-file, which is a gzipped text file. If I try spark.read.text('some-file'), it will return a bunch of gibberish since it doesn't know that the file is gzipped. I'm looking to manually tell spark the fil...
- 8227 Views
- 2 replies
- 0 kudos
- 0 kudos
The community is field for the approval of the terms. The struggle of a great site is recommend for the norms. The value is suggested for the top of the vital paths for the finding members.
- 0 kudos
- 7890 Views
- 4 replies
- 3 kudos
- 7890 Views
- 4 replies
- 3 kudos
- 3 kudos
probably it is not in PATH, you can add it to PATH as described here https://ganeshchandrasekaran.com/how-to-install-databricks-cli-and-get-the-path-of-databricks-executable-on-windows-74f83040dde7
- 3 kudos
- 20516 Views
- 9 replies
- 2 kudos
Setup Git Integration via REST API
We are currently setting up CI/CD for our Databricks workspace using Databricks Repos following the approach described in the offical docs: https://docs.databricks.com/repos.html#best-practices-for-integrating-databricks-repos-with-cicd-workflowsObvi...
- 20516 Views
- 9 replies
- 2 kudos
- 3735 Views
- 1 replies
- 1 kudos
Resolved! concat_ws() throws AnalysisException when too many columns are supplied
Hi,i'm using concat_ws in scala to calculate a checksum for the dataframe, i.e.:df.withColumn("CHECKSUM", sha2(functions.concat_ws("", dataframe.columns.map(col): _*), 512))I have one example here with just 24 columns that already throws the followin...
- 3735 Views
- 1 replies
- 1 kudos
- 1 kudos
at least one of column names can have some strange char, whitespace or something,or at least one of column type is not compatible (for example StructType)you can separate your code to two or more steps. First generate list of columns as some variable...
- 1 kudos
- 6678 Views
- 2 replies
- 2 kudos
Resolved! How do i changes the fonts and color of the title of widget and the background color of widget?
Currently there is no documentation on how I can change the fonts and background color of widget? Is there a way to do so?
- 6678 Views
- 2 replies
- 2 kudos
- 2 kudos
I saw on roadmap presentation that in future there will be more widget options but for now is like @Kavya Manohar Parag​ said.
- 2 kudos
- 18877 Views
- 6 replies
- 5 kudos
- 18877 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Tony Zhou​ , Were you able to run this after generating a new token?
- 5 kudos
- 4271 Views
- 2 replies
- 3 kudos
Resolved! a user has been deleted from databricks workspace . Is there any way to find who deleted the user?
a user has been deleted from databricks workspace . Is there any way to find who deleted the user?
- 4271 Views
- 2 replies
- 3 kudos
- 3 kudos
To do that you need to have enabled audit logs (if event already happened and it was not "on" I am afraid now it is too late).For Azure https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logsFor A...
- 3 kudos
- 1375 Views
- 0 replies
- 0 kudos
Image Display in Dockerized Cluster
I am using a docker recipe for configuring my databricks cluster. It is working fine for everything else however when I tried to display any image data using any python utility such as matplotlib, PIL or Opencv etc. the image does not get displayed o...
- 1375 Views
- 0 replies
- 0 kudos
- 10392 Views
- 4 replies
- 15 kudos
Resolved! What's the best way to run a databricks notebook from AWS Lambda ?
I have a trigger in lambda that gets triggered when a new file arrives in S3. I want this file to be straightaway processed using a notebook to Upsert all the data into a delta table.I'm looking for a solution with minimum latency.
- 10392 Views
- 4 replies
- 15 kudos
- 15 kudos
There are two possible solution:autoloader/cloudfiles, better with "File notification" queue to avoid unnecessary scans,ORfrom lambda sending post request to /api/2.1/jobs/run-nowAdditionally in both solution it is important to have private link and...
- 15 kudos
- 7235 Views
- 4 replies
- 3 kudos
Resolved! pySpark Dataframe to DeepLearning model
I have a large time series with many measuring stations recording the same 5 data (Temperature, Humidity, etc.) I want to predict a future moment with a time series model, for which I pass the data from all the measuring stations to the Deep Learning...
- 7235 Views
- 4 replies
- 3 kudos
- 3 kudos
df.groupBy("date").pivot("Node").agg(first("Temp"))It is converting to classic crosstable so pivot will help. Example above.
- 3 kudos
- 8086 Views
- 3 replies
- 1 kudos
Resolved! Unable to clone GitLab Enterprise Edition repo in Databricks
Below are the steps that I followed. I still get an error message.Create a repo in gitlab enterprise editionIn GitLab, create a personal access token that allows access to your repositories ( with read_repository and write_repository permissions)Save...
- 8086 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Sarvagna Mahakali​ the repository which you are trying to add might be behind the VPN, our service cannot access it since it has no access to the VPN network.You may need the Enterprise Git / VPC to connect to the repository.Kindly check and let...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
