- 23404 Views
- 1 replies
- 2 kudos
Creating permanent views from dataframes?
Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...
- 23404 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...
- 2 kudos
- 3538 Views
- 2 replies
- 2 kudos
Simba ODBC datetime with millisecond overflows
Hello,Using odbc 2.6.24.1041-2 for Linux, when inserting rows with milliseconds precision date (e.g. 2022-07-03 13:57:48.500) precision I get:2022/07/03 14:41:19 SQLExecute: {22008} [Simba][Support] (40520) Datetime field overflow resulting from inva...
- 3538 Views
- 2 replies
- 2 kudos
- 2 kudos
I was passing a string (e.g. '2022-07-03 13:57:48.500') to the Golang SQL driver which is not working if the ms part is specified, but otherwise it works (e.g. '2022-07-03 13:57:48'). Passing a native Golang time.Time seems to work for timestamps wit...
- 2 kudos
- 1286 Views
- 0 replies
- 5 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks expe...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks experts! Our next events are scheduled for July 13th & July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GM...
- 1286 Views
- 0 replies
- 5 kudos
- 1887 Views
- 0 replies
- 0 kudos
Unexpected behaviour when creating multiple Account-Level Accounts
We've had a qustion regarding possibly unexpected behaviour when creating multiple accounts on the account-level on https://accounts.cloud.databricks.com/.Short Version:It's possible to create multiple accounts with different letter-cases on https://...
- 1887 Views
- 0 replies
- 0 kudos
- 5389 Views
- 3 replies
- 1 kudos
Resolved! ipywidgets not working in DBR 11.0 on Community Edition
I'm looking forward to using ipywidgets which should be working in DBR 11.0 as they provide more options when creating a notebook UI. I saw that DBR 11.0 is available as of yesterday so I created a test cluster in the Databricks Community Edition ju...
- 5389 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Wayne Deleersnyder​ I was able to import ipywidgets in DBR 11.0. As you can see in the output below. The slider is visibleYou are facing this issue probably because the community edition has limited access. To get all the features you should at l...
- 1 kudos
- 1931 Views
- 3 replies
- 2 kudos
pd.read_csv failed
https://i.imgur.com/PAGzSr9.png
- 1931 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks for your kind reply:Below works for me:https://imgur.com/BmMzatIBut why, as you mentioned, using the classic path, below does not work?https://imgur.com/Ba1a4Iv
- 2 kudos
- 1525 Views
- 0 replies
- 0 kudos
How to make a paper airplane
How to make a paper airplane
- 1525 Views
- 0 replies
- 0 kudos
- 2345 Views
- 2 replies
- 0 kudos
DBrick workspace URL block outside the corporate network?
As per security concern, need to restrict/block the dbricks workspace url outside the corporate network. Tried below ip access list, it able to restrict only user login access out the corporate network but still the workspace id url is live outside t...
- 2345 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @as999​ Hope everything is going great!Does @Atanu Sarkar​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. We'...
- 0 kudos
- 5891 Views
- 3 replies
- 1 kudos
How do you install R packages?
I've tried this, but it doesn't appear to be working: https://community.databricks.com/s/question/0D53f00001GHVX1CAP/unable-to-install-sf-and-rgeos-r-packages-on-the-clusterWhen I run the following after that init script, I receive an error.library(r...
- 5891 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey there @Christopher Flach​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 1 kudos
- 7499 Views
- 5 replies
- 0 kudos
Resolved! A void column was created after connecting to cosmos
Hi everyone, I have connected to Cosmos using this tutorial https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3_2-12/Samples/DatabricksLiveContainerMigrationAfter creating a table using a simple SQL command:CREATE TA...
- 7499 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey there @Joel iemma​ Hope all is well! Just wanted to check in if you would be happy to mark an answer as best for us, please? It would be really helpful for the other members too.Cheers!
- 0 kudos
- 4640 Views
- 2 replies
- 0 kudos
Resolved! JDBC Connectivity via workspace url when No Public IP selected.
Hi All, I think I might be missing something in regard to No Pubic IP Clusters. I have set this option on a workspace (Azure) and setup the appropriate subnets. To my surprise, when I went to setup a JDBC connection to the cluster the JDBC connec...
- 4640 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey there @Ashley Betts​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
- 0 kudos
- 24545 Views
- 18 replies
- 13 kudos
Resolved! HELP! Converting GZ JSON to Delta causes massive CPU spikes and ETL's take days!
Hi there,I was wondering if I could get your advise.We would like to create a bronze delta table using GZ JSON data stored in S3 but each time we attempt to read and write it our clusters CPU spikes to 100%. We are not doing any transformations but s...
- 24545 Views
- 18 replies
- 13 kudos
- 13 kudos
Hi Kaniz,Thanks for the note and thank you everyone for the suggestions and help. @Joseph Kambourakis​ I aded your suggestion to our load but I did not see any change in how our data loads or the time it takes to load data. I've done some additional ...
- 13 kudos
- 11993 Views
- 3 replies
- 4 kudos
Resolved! Recommended way to integrate MongoDB as a streaming source
Current state:Data is stored in MongoDB Atlas which is used extensively by all servicesData lake is hosted in same AWS region and connected to MongoDB over private link Requirements:Streaming pipelines that continuously ingest, transform/analyze and ...
- 11993 Views
- 3 replies
- 4 kudos
- 4 kudos
Another option if you'd like to use Spark as the ingestion is to use the new Spark Connector V10.0 which support Spark Structured Streaming. https://www.mongodb.com/developer/languages/python/streaming-data-apache-spark-mongodb/. If you use Kafka, th...
- 4 kudos
- 2998 Views
- 2 replies
- 0 kudos
Resolved! Error during setup of cluster
Unexpected Launch Failure: An unexpected error was encountered while setting up the cluster. Please retry and contact Azure Databricks if the problem persists. Internal error message: Timeout while placing node.
- 2998 Views
- 2 replies
- 0 kudos
- 0 kudos
Ensure that CNO$ account has Full Control on the CNO and The computers container;Add CNO$ account (CNO computer object) in Local Admins group;Finally, add CNO$ in Domain Admins group.Regards,Willjoe
- 0 kudos
- 98513 Views
- 6 replies
- 13 kudos
Can databricks be used locally to learn it or is it cloud only
I'm tired of telling clients or referrals I don't know databricks but it seems like the only option is to have a big AWS account and then use databricks on that data. Can I download it locally for training, upskilling with python or is it only for cl...
- 98513 Views
- 6 replies
- 13 kudos
- 13 kudos
Thanks for linking directly to the docker image @Hubert Dudek​ ! And thanks for the info @Prabakar Ammeappin​ and @Amit Nainawati​ @Andrew Schell​ Let us know if you have more questions! If not, choose a best answer in this thread and let us know how...
- 13 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
