- 6072 Views
- 3 replies
- 6 kudos
Resolved! Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?
We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?
- 6072 Views
- 3 replies
- 6 kudos
- 6 kudos
Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time. you can have for example following configuration:In Azure you can have 2 databricks workspace,cluster in every workspace can have env variable is...
- 6 kudos
- 9500 Views
- 5 replies
- 1 kudos
Resolved! Quick way to know delta table is zordered
Hello,I created a delta table table using SQL and specifying the partitioning and zorder strategy. I then loaded data into it for the first time by doing a write as delta with mode of append and save as table. However, I don’t know of a way to verify...
- 9500 Views
- 5 replies
- 1 kudos
- 1 kudos
If there is no data then lines 10 and 11 will not have any impact. I am assuming that line (1-5) is creating an empty table but the actual load is happening when you do df.write operation. Also delta.autoOptimize.autoCompact will not trigger the z-or...
- 1 kudos
- 5343 Views
- 2 replies
- 4 kudos
Resolved! Error in accessing Delta Tables
I'm getting attached error in accessing delta lake tables in the data bricks workspaceSummary of error: Could not connect to md1n4trqmokgnhr.csnrqwqko4ho.ap-southeast-1.rds.amazonaws.com:3306 : Connection resetAttached detailed error
- 5343 Views
- 2 replies
- 4 kudos
- 4 kudos
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to md1n4trqmokgnhr.csnrqwqko4ho.ap-southeast-1.rds.amazonaws.com:3306 : Connection reset at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:...
- 4 kudos
- 8914 Views
- 8 replies
- 4 kudos
Resolved! How to write Change Data from Delta Lake to aws dynamodb
Is there some direct way to write data from DeltaLake to AWS DynamoDB.If there is none, Is there any way to do the same.
- 8914 Views
- 8 replies
- 4 kudos
- 4 kudos
Hi @Harikrishnan P H​ ,Did @Werner Stinckens​ reply help you to resolved your issue? if yes, please mark it as best. if not, please let us know.
- 4 kudos
- 8676 Views
- 5 replies
- 2 kudos
Resolved! Databricks Data Type Conversion error
In databricks while writing data to curated layer, see error - Failed to execute user defined function (Double => decimal(38,18)). Does anyone know if faced such issue and how to resolve it.
- 8676 Views
- 5 replies
- 2 kudos
- 2 kudos
What happens if you explicitly cast it?I remember having such issues with implicit casting when goin from spark 2.x to 3.x, but these were solved by using explicit casting (not round()).
- 2 kudos
- 1170 Views
- 1 replies
- 2 kudos
The Next Databricks Office HoursOur next Office Hours session is scheduled for March 23 2022 - 8:00 am PDTÂ Do you have questions about how to set up o...
The Next Databricks Office HoursOur next Office Hours session is scheduled for March 23 2022 - 8:00 am PDTDo you have questions about how to set up or use Databricks? Do you want to get best practices for deploying your use case or tips on data archi...
- 1170 Views
- 1 replies
- 2 kudos
- 2 kudos
Signed in!
- 2 kudos
- 2902 Views
- 1 replies
- 0 kudos
- 2902 Views
- 1 replies
- 0 kudos
- 0 kudos
@Bhagwan Chaubey​ May be you can give this a try, if this is a Blob Storage account.https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windowsFor Datalake storage, please try belowhttps://do...
- 0 kudos
- 9333 Views
- 4 replies
- 3 kudos
Resolved! Writing Spark data frame to ADLS is taking Huge time when Data Frame is of Text data.
Spark data frame with text data when schema is in Struct type spark is taking too much time to write / save / push data to ADLS or SQL Db or download as csv.
- 9333 Views
- 4 replies
- 3 kudos
- 3 kudos
@shiva Santosh​ Have to checked the count of the dataframe that you are trying to save to ADLS?As @Joseph Kambourakis​ mentioned the explode can result in 1-many rows, better to check data frame count and see if Spark OOMs in the workspace.
- 3 kudos
- 3115 Views
- 1 replies
- 2 kudos
github.com
Do I always need to manually invite members of my AAD tenant to ADB workspace, if I don't have SCIM integration configured?EDIT: solved, it works when you go through Azure Portal and get in with "Launch Workspace" button on the ADB resource overview ...
- 3115 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @pawelmitrus​ For users with Owner or Contributor roles, they should click on the "Launch Workspace" button in the Azure portal. For other users they should be explicitly granted access to the workspace to be able to login.Regards,Arvind
- 2 kudos
- 2901 Views
- 1 replies
- 4 kudos
Resolved! Databricks Community: Cluster Terminated Reason: Unexpected Launch Failure
Hi,I've been encountering the following error when I try to start a cluster, but the status page says everything is fine. Is something happening or are there other steps I can try?Time2022-03-13 14:40:51 EDTMessageCluster terminated.Reason:Unexpected...
- 2901 Views
- 1 replies
- 4 kudos
- 4 kudos
Hi @Rachel Kelley​ We have some internal service interruptions due to which we had this issue. Our engineering has applied the fix and the cluster startup works as expected. Sincerely apologies for the inconvenience caused here.Regards,Darshan
- 4 kudos
- 7636 Views
- 2 replies
- 0 kudos
How to read a compressed file in spark if the filename does not include the file extension for that compression format?
For example, let's say I have a file called some-file, which is a gzipped text file. If I try spark.read.text('some-file'), it will return a bunch of gibberish since it doesn't know that the file is gzipped. I'm looking to manually tell spark the fil...
- 7636 Views
- 2 replies
- 0 kudos
- 0 kudos
The community is field for the approval of the terms. The struggle of a great site is recommend for the norms. The value is suggested for the top of the vital paths for the finding members.
- 0 kudos
- 6407 Views
- 4 replies
- 3 kudos
- 6407 Views
- 4 replies
- 3 kudos
- 3 kudos
probably it is not in PATH, you can add it to PATH as described here https://ganeshchandrasekaran.com/how-to-install-databricks-cli-and-get-the-path-of-databricks-executable-on-windows-74f83040dde7
- 3 kudos
- 18171 Views
- 9 replies
- 2 kudos
Setup Git Integration via REST API
We are currently setting up CI/CD for our Databricks workspace using Databricks Repos following the approach described in the offical docs: https://docs.databricks.com/repos.html#best-practices-for-integrating-databricks-repos-with-cicd-workflowsObvi...
- 18171 Views
- 9 replies
- 2 kudos
- 3250 Views
- 1 replies
- 1 kudos
Resolved! concat_ws() throws AnalysisException when too many columns are supplied
Hi,i'm using concat_ws in scala to calculate a checksum for the dataframe, i.e.:df.withColumn("CHECKSUM", sha2(functions.concat_ws("", dataframe.columns.map(col): _*), 512))I have one example here with just 24 columns that already throws the followin...
- 3250 Views
- 1 replies
- 1 kudos
- 1 kudos
at least one of column names can have some strange char, whitespace or something,or at least one of column type is not compatible (for example StructType)you can separate your code to two or more steps. First generate list of columns as some variable...
- 1 kudos
- 6008 Views
- 2 replies
- 2 kudos
Resolved! How do i changes the fonts and color of the title of widget and the background color of widget?
Currently there is no documentation on how I can change the fonts and background color of widget? Is there a way to do so?
- 6008 Views
- 2 replies
- 2 kudos
- 2 kudos
I saw on roadmap presentation that in future there will be more widget options but for now is like @Kavya Manohar Parag​ said.
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
