- 1607 Views
- 1 replies
- 2 kudos
Take advantage of the Data + AI Summit Virtual Experience next week! Data + AI Summit is just a few days away! With data professionals from 155+ count...
Take advantage of the Data + AI Summit Virtual Experience next week!Data + AI Summit is just a few days away! With data professionals from 155+ countries already registered, this is truly the premier event for the global data, analytics and AI commun...
- 1607 Views
- 1 replies
- 2 kudos
- 2 kudos
Thank you for informing highlights of the upcoming sessions.
- 2 kudos
- 896 Views
- 1 replies
- 0 kudos
- 896 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Muhammad Umer​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2344 Views
- 2 replies
- 4 kudos
docs.databricks.com
Databricks new Release : Full-page workspace browser includes ReposDatabricks plans to enable the full-page workspace browser experience that unifies Workspace and Repos by default. You’ll browse content in Databricks Repos alongside your workspace c...
- 2344 Views
- 2 replies
- 4 kudos
- 4681 Views
- 4 replies
- 2 kudos
Delta file question
Hi! Im using Autoloader to ingest Binary files into delta format. I have 7 binary files but delta generate 3 files and the format is part-0000, part-0001... Why generate this files with format part-000...
- 4681 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Alejandro Piury Pinzón​ We haven't heard from you since the last response from @Lakshay Goel​ r​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...
- 2 kudos
- 2240 Views
- 1 replies
- 0 kudos
Issues loading files csv files that contain BOM (Byte Order Mark) character
I keep getting and error when creating dataframe or steam from certain CSV files where the header contains BOM (Byte Order Mark) character This is the error message:AnalysisException: [RequestId=e09c7c8d-2399-4d6a-84ae-216e6a9f8f6e ErrorClass=INVALI...
- 2240 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mohamed harraz​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 1443 Views
- 1 replies
- 0 kudos
CDM connector for Spark can't connect to Azure storage account
Hello,I'm trying to use the CDM connector for Spark, but I can't connecto to the Azure storage account when using the connector. I mounted a container of storage account with a SAS-token. When I'm trying to read CDM data from a (mounted) storage acco...
- 1443 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Martijn de Bruijn​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 2769 Views
- 3 replies
- 2 kudos
[AWS] How do you replace the Account Admin?
I need to remove an older admin that previously set up the Databricks Account. However, I get an error (even through I am also an Account Admin).How do I replace a prior account admin? Or at least remove their admin status and/or disable the accoun...
- 2769 Views
- 3 replies
- 2 kudos
- 2 kudos
I escalated via my Databricks rep yesterday and got an answer that seemed to be along the lines that "something is wrong here". He is going to try to find out internally and possibly work with the product development folks to come up with a solution...
- 2 kudos
- 3174 Views
- 2 replies
- 1 kudos
about .crc file in delta transaction log
why .crc file will create along with delta log files
- 3174 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Lakshmi devi​ , crc file is basically a checksum file that contains the stats for the respective version file. It is used for snapshot verification in the backend.
- 1 kudos
- 6351 Views
- 2 replies
- 1 kudos
Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...
- 6351 Views
- 2 replies
- 1 kudos
- 1 kudos
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- 1 kudos
- 5262 Views
- 2 replies
- 2 kudos
getContext() in dbutils.notebook not working in DBR 11.2 10.4 LTS Shared Mode It's also working in no isolation Mode in DBR 11.2
We are trying to fetch notebook context in our Job logging workflow.current_context = dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson()We were able to access this in DBR 10.4 custom mode but in DBR 10.4 & 11.2 (Shared Mode) w...
- 5262 Views
- 2 replies
- 2 kudos
- 2 kudos
See also: https://community.databricks.com/s/question/0D58Y00009t95NHSAY/unity-catalog-shared-access-mode-dbutilsnotebookentrypointgetcontext-not-whitelisted
- 2 kudos
- 12553 Views
- 6 replies
- 0 kudos
Resolved! How to copy the content of a repos, in the workspace/shared, automatically, daily?
How to copy the content of a repos, in the workspace/shared, automatically, daily?Purpose here is to bring some notebooks, in shared, available to all workspace users, without requesting users to use REPOS.
- 12553 Views
- 6 replies
- 0 kudos
- 0 kudos
Alternatively, you can schedule a notebook with the following cell:%sh cp -r /Workspace/Repos/username/repo_name /Workspace/SharedPlease note that you'll have to update (i.e. pull) the repo manually if you've updated it somewhere else (although you c...
- 0 kudos
- 984 Views
- 0 replies
- 1 kudos
Connect to warehouse vs connect to compute/cluster
When I create a warehouse, there are connection details which I was able to successfully use, but there are also connection details when I navigate in the console to Compute->configuration->Advanced options-> JDBC/ODBC What is the difference between ...
- 984 Views
- 0 replies
- 1 kudos
- 7464 Views
- 3 replies
- 0 kudos
Linear Regression HELP! Pickle + Broadcast Variable Error
Hi there,I need some help with this example. We're trying to create a linearRegression model that can parallelize for thousands of symbols per date. When we run this we get a picklingError Any suggestions would be much appreciated!KError:PicklingErro...
- 7464 Views
- 3 replies
- 0 kudos
- 11960 Views
- 3 replies
- 9 kudos
Resolved! Delta save timestamp as timestamp with time zone
Hello! I have the following problem. I want to save the delta table and that table contains timestamp columns, but when I try to write that table with spark the timestamp columns become timestamp with the time zone. This is a problem in my case becau...
- 11960 Views
- 3 replies
- 9 kudos
- 9 kudos
Hi @Hubert Dudek​ ,When you have time, I describe my problem. Can you please check?
- 9 kudos
- 2743 Views
- 2 replies
- 0 kudos
Running a K-means (.fit) gives error:Params must be either a param map or a list/tuple of param maps but got %s." % type(params)
am running a k-means algorithm. My feature are DoubleType and have no nulls, but I get : raise TypeError("Params must be either a param map or a list/tuple of param maps but got %s." % type(params). Anyone have any idea how to solve this?File /datab...
- 2743 Views
- 2 replies
- 0 kudos
- 0 kudos
I found the answer just by trying several things, although I do not understand exactly what the problem was. All I had to do was to cache the input data before fitting the model:assemble=VectorAssembler(inputCols=columns_input, outputCol='features')...
- 0 kudos