- 3377 Views
- 1 replies
- 2 kudos
github.com
Do I always need to manually invite members of my AAD tenant to ADB workspace, if I don't have SCIM integration configured?EDIT: solved, it works when you go through Azure Portal and get in with "Launch Workspace" button on the ADB resource overview ...
- 3377 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @pawelmitrus​ For users with Owner or Contributor roles, they should click on the "Launch Workspace" button in the Azure portal. For other users they should be explicitly granted access to the workspace to be able to login.Regards,Arvind
- 2 kudos
- 3214 Views
- 1 replies
- 4 kudos
Resolved! Databricks Community: Cluster Terminated Reason: Unexpected Launch Failure
Hi,I've been encountering the following error when I try to start a cluster, but the status page says everything is fine. Is something happening or are there other steps I can try?Time2022-03-13 14:40:51 EDTMessageCluster terminated.Reason:Unexpected...
- 3214 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Rachel Kelley​ We have some internal service interruptions due to which we had this issue. Our engineering has applied the fix and the cluster startup works as expected. Sincerely apologies for the inconvenience caused here.Regards,Darshan
- 4 kudos
- 8074 Views
- 2 replies
- 0 kudos
How to read a compressed file in spark if the filename does not include the file extension for that compression format?
For example, let's say I have a file called some-file, which is a gzipped text file. If I try spark.read.text('some-file'), it will return a bunch of gibberish since it doesn't know that the file is gzipped. I'm looking to manually tell spark the fil...
- 8074 Views
- 2 replies
- 0 kudos
- 0 kudos
The community is field for the approval of the terms. The struggle of a great site is recommend for the norms. The value is suggested for the top of the vital paths for the finding members.
- 0 kudos
- 7388 Views
- 4 replies
- 3 kudos
- 7388 Views
- 4 replies
- 3 kudos
- 3 kudos
probably it is not in PATH, you can add it to PATH as described here https://ganeshchandrasekaran.com/how-to-install-databricks-cli-and-get-the-path-of-databricks-executable-on-windows-74f83040dde7
- 3 kudos
- 19761 Views
- 9 replies
- 2 kudos
Setup Git Integration via REST API
We are currently setting up CI/CD for our Databricks workspace using Databricks Repos following the approach described in the offical docs: https://docs.databricks.com/repos.html#best-practices-for-integrating-databricks-repos-with-cicd-workflowsObvi...
- 19761 Views
- 9 replies
- 2 kudos
- 3612 Views
- 1 replies
- 1 kudos
Resolved! concat_ws() throws AnalysisException when too many columns are supplied
Hi,i'm using concat_ws in scala to calculate a checksum for the dataframe, i.e.:df.withColumn("CHECKSUM", sha2(functions.concat_ws("", dataframe.columns.map(col): _*), 512))I have one example here with just 24 columns that already throws the followin...
- 3612 Views
- 1 replies
- 1 kudos
- 1 kudos
at least one of column names can have some strange char, whitespace or something,or at least one of column type is not compatible (for example StructType)you can separate your code to two or more steps. First generate list of columns as some variable...
- 1 kudos
- 6528 Views
- 2 replies
- 2 kudos
Resolved! How do i changes the fonts and color of the title of widget and the background color of widget?
Currently there is no documentation on how I can change the fonts and background color of widget? Is there a way to do so?
- 6528 Views
- 2 replies
- 2 kudos
- 2 kudos
I saw on roadmap presentation that in future there will be more widget options but for now is like @Kavya Manohar Parag​ said.
- 2 kudos
- 18457 Views
- 6 replies
- 5 kudos
- 18457 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @Tony Zhou​ , Were you able to run this after generating a new token?
- 5 kudos
- 4084 Views
- 2 replies
- 3 kudos
Resolved! a user has been deleted from databricks workspace . Is there any way to find who deleted the user?
a user has been deleted from databricks workspace . Is there any way to find who deleted the user?
- 4084 Views
- 2 replies
- 3 kudos
- 3 kudos
To do that you need to have enabled audit logs (if event already happened and it was not "on" I am afraid now it is too late).For Azure https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logsFor A...
- 3 kudos
- 1298 Views
- 0 replies
- 0 kudos
Image Display in Dockerized Cluster
I am using a docker recipe for configuring my databricks cluster. It is working fine for everything else however when I tried to display any image data using any python utility such as matplotlib, PIL or Opencv etc. the image does not get displayed o...
- 1298 Views
- 0 replies
- 0 kudos
- 10092 Views
- 4 replies
- 15 kudos
Resolved! What's the best way to run a databricks notebook from AWS Lambda ?
I have a trigger in lambda that gets triggered when a new file arrives in S3. I want this file to be straightaway processed using a notebook to Upsert all the data into a delta table.I'm looking for a solution with minimum latency.
- 10092 Views
- 4 replies
- 15 kudos
- 15 kudos
There are two possible solution:autoloader/cloudfiles, better with "File notification" queue to avoid unnecessary scans,ORfrom lambda sending post request to /api/2.1/jobs/run-nowAdditionally in both solution it is important to have private link and...
- 15 kudos
- 6776 Views
- 4 replies
- 3 kudos
Resolved! pySpark Dataframe to DeepLearning model
I have a large time series with many measuring stations recording the same 5 data (Temperature, Humidity, etc.) I want to predict a future moment with a time series model, for which I pass the data from all the measuring stations to the Deep Learning...
- 6776 Views
- 4 replies
- 3 kudos
- 3 kudos
df.groupBy("date").pivot("Node").agg(first("Temp"))It is converting to classic crosstable so pivot will help. Example above.
- 3 kudos
- 7699 Views
- 3 replies
- 1 kudos
Resolved! Unable to clone GitLab Enterprise Edition repo in Databricks
Below are the steps that I followed. I still get an error message.Create a repo in gitlab enterprise editionIn GitLab, create a personal access token that allows access to your repositories ( with read_repository and write_repository permissions)Save...
- 7699 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Sarvagna Mahakali​ the repository which you are trying to add might be behind the VPN, our service cannot access it since it has no access to the VPN network.You may need the Enterprise Git / VPC to connect to the repository.Kindly check and let...
- 1 kudos
- 32145 Views
- 1 replies
- 3 kudos
Resolved! dataframe - cast string to decimal when encountering zeros returns OE-16
The user is trying to cast string to decimal when encountering zeros. The cast function displays the '0' as '0E-16'. could you please let us know your thoughts on whether 0s can be displayed as 0s?from pyspark.sql import functions as F df = spark.s...
- 32145 Views
- 1 replies
- 3 kudos
- 3 kudos
If the scale of decimal type is greater than 6, scientific notation kicks in hence seeing 0E-16.This behavior is described in the existing OSS spark issue - https://issues.apache.org/jira/browse/SPARK-25177Kindly cast the column to a decimal type les...
- 3 kudos
- 6753 Views
- 7 replies
- 2 kudos
Resolved! Long time turning on another notebook
Hello, I want to run some notebooks from notebook "A". And regardless of the contents of the some notebook, it is run for a long time (20 seconds). It is constans value and I do not know why it takes so long. I tried run simple notebook with one in...
- 6753 Views
- 7 replies
- 2 kudos
- 2 kudos
Okay I am not able to set the same session for the both notebooks (parent and children).So my result is to use %run ./notebook_name .I put all the code to functions and now I can use them.Example:# Children notebook def do_something(param1, param2): ...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
